2022-03-17T19:05:52.4267884Z Requested labels: linux.8xlarge.nvidia.gpu 2022-03-17T19:05:52.4267990Z Job defined at: pytorch/pytorch/.github/workflows/generated-linux-xenial-cuda11.3-py3.7-gcc7.yml@refs/heads/master 2022-03-17T19:05:52.4268022Z Waiting for a runner to pick up this job... 2022-03-17T19:06:12.7802540Z Job is about to start running on the runner: i-038ed9ad29e72a29d (repository) 2022-03-17T19:06:17.3785955Z Current runner version: '2.288.1' 2022-03-17T19:06:17.3794020Z Runner name: 'i-038ed9ad29e72a29d' 2022-03-17T19:06:17.3794679Z Runner group name: 'Default' 2022-03-17T19:06:17.3795448Z Machine name: 'ip-10-0-4-102' 2022-03-17T19:06:17.3798405Z ##[group]GITHUB_TOKEN Permissions 2022-03-17T19:06:17.3799328Z Actions: write 2022-03-17T19:06:17.3799711Z Checks: write 2022-03-17T19:06:17.3800123Z Contents: write 2022-03-17T19:06:17.3800853Z Deployments: write 2022-03-17T19:06:17.3801249Z Discussions: write 2022-03-17T19:06:17.3801682Z Issues: write 2022-03-17T19:06:17.3802047Z Metadata: read 2022-03-17T19:06:17.3802470Z Packages: write 2022-03-17T19:06:17.3802888Z Pages: write 2022-03-17T19:06:17.3803336Z PullRequests: write 2022-03-17T19:06:17.3803783Z RepositoryProjects: write 2022-03-17T19:06:17.3804276Z SecurityEvents: write 2022-03-17T19:06:17.3804716Z Statuses: write 2022-03-17T19:06:17.3805138Z ##[endgroup] 2022-03-17T19:06:17.3809509Z Secret source: Actions 2022-03-17T19:06:17.3810303Z Prepare workflow directory 2022-03-17T19:06:17.5354895Z Prepare all required actions 2022-03-17T19:06:17.5582501Z Getting action download info 2022-03-17T19:06:17.7759654Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:1ecffedb1e192a50aa67dba2f0e048e5d3bfa144) 2022-03-17T19:06:18.0645191Z Download action repository 'zhouzhuojie/checkout@05b13c9a0d21f08f6d5e64a1d5042246d13619d9' (SHA:05b13c9a0d21f08f6d5e64a1d5042246d13619d9) 2022-03-17T19:06:18.2539619Z Download action repository 'nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a' (SHA:71062288b76e2b6214ebde0e673ce0de1755740a) 2022-03-17T19:06:18.4541999Z Download action repository 'seemethere/download-artifact-s3@0504774707cbc8603d7dca922e8026eb8bf3b47b' (SHA:0504774707cbc8603d7dca922e8026eb8bf3b47b) 2022-03-17T19:06:18.7167680Z Download action repository 'seemethere/upload-artifact-s3@v3' (SHA:9d298f4a34a19beb76876fe16bcd4aa6389bd42c) 2022-03-17T19:06:19.2046301Z ##[group]Run set -euo pipefail 2022-03-17T19:06:19.2046770Z set -euo pipefail 2022-03-17T19:06:19.2047061Z function get_ec2_metadata() { 2022-03-17T19:06:19.2047383Z  # Pulled from instance metadata endpoint for EC2 2022-03-17T19:06:19.2047864Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2022-03-17T19:06:19.2048250Z  category=$1 2022-03-17T19:06:19.2048556Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2022-03-17T19:06:19.2048846Z } 2022-03-17T19:06:19.2049111Z echo "ami-id: $(get_ec2_metadata ami-id)" 2022-03-17T19:06:19.2049432Z echo "instance-id: $(get_ec2_metadata instance-id)" 2022-03-17T19:06:19.2049796Z echo "instance-type: $(get_ec2_metadata instance-type)" 2022-03-17T19:06:19.2050108Z echo "system info $(uname -a)" 2022-03-17T19:06:19.2068592Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T19:06:19.2068896Z env: 2022-03-17T19:06:19.2069189Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:06:19.2069685Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:06:19.2070158Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:06:19.2070564Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:06:19.2070903Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:06:19.2071128Z IN_CI: 1 2022-03-17T19:06:19.2071327Z IS_GHA: 1 2022-03-17T19:06:19.2071532Z IN_WHEEL_TEST: 1 2022-03-17T19:06:19.2071818Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:06:19.2072182Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:06:19.2072502Z PR_LABELS: [] 2022-03-17T19:06:19.2073015Z GITHUB_TOKEN: *** 2022-03-17T19:06:19.2073279Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:06:19.2073558Z PR_NUMBER: 2022-03-17T19:06:19.2073808Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:19.2074255Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:06:19.2074504Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:06:19.2074993Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:06:19.2075524Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:06:19.2075990Z TEST_CONFIG: distributed 2022-03-17T19:06:19.2076231Z SHARD_NUMBER: 1 2022-03-17T19:06:19.2076503Z NUM_TEST_SHARDS: 1 2022-03-17T19:06:19.2076725Z PR_BODY: 2022-03-17T19:06:19.2076962Z ##[endgroup] 2022-03-17T19:06:19.2369677Z ami-id: ami-096198a0bccc6bad4 2022-03-17T19:06:19.2393214Z instance-id: i-038ed9ad29e72a29d 2022-03-17T19:06:19.2456105Z instance-type: g3.8xlarge 2022-03-17T19:06:19.2464815Z system info Linux ip-10-0-4-102.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux 2022-03-17T19:06:19.2634390Z ##[group]Run if systemctl is-active --quiet docker; then 2022-03-17T19:06:19.2634845Z if systemctl is-active --quiet docker; then 2022-03-17T19:06:19.2635173Z  echo "Docker daemon is running..."; 2022-03-17T19:06:19.2635422Z else 2022-03-17T19:06:19.2635729Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2022-03-17T19:06:19.2636013Z fi 2022-03-17T19:06:19.2647738Z shell: /usr/bin/bash -e {0} 2022-03-17T19:06:19.2647970Z env: 2022-03-17T19:06:19.2648269Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:06:19.2648762Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:06:19.2649248Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:06:19.2649671Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:06:19.2650011Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:06:19.2650271Z IN_CI: 1 2022-03-17T19:06:19.2650470Z IS_GHA: 1 2022-03-17T19:06:19.2650690Z IN_WHEEL_TEST: 1 2022-03-17T19:06:19.2650976Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:06:19.2651366Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:06:19.2651678Z PR_LABELS: [] 2022-03-17T19:06:19.2652130Z GITHUB_TOKEN: *** 2022-03-17T19:06:19.2652389Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:06:19.2652633Z PR_NUMBER: 2022-03-17T19:06:19.2652886Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:19.2653164Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:06:19.2653416Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:06:19.2653914Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:06:19.2654453Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:06:19.2654757Z TEST_CONFIG: distributed 2022-03-17T19:06:19.2655001Z SHARD_NUMBER: 1 2022-03-17T19:06:19.2655226Z NUM_TEST_SHARDS: 1 2022-03-17T19:06:19.2655455Z PR_BODY: 2022-03-17T19:06:19.2655663Z ##[endgroup] 2022-03-17T19:06:19.2708165Z Docker daemon is running... 2022-03-17T19:06:19.2730143Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-03-17T19:06:19.2730668Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") 2022-03-17T19:06:19.2730995Z retry () { 2022-03-17T19:06:19.2731253Z  "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") 2022-03-17T19:06:19.2731510Z } 2022-03-17T19:06:19.2732084Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \ 2022-03-17T19:06:19.2732540Z  --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2022-03-17T19:06:19.2744658Z shell: /usr/bin/bash -e {0} 2022-03-17T19:06:19.2744897Z env: 2022-03-17T19:06:19.2745200Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:06:19.2745695Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:06:19.2746343Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:06:19.2746750Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:06:19.2747101Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:06:19.2747334Z IN_CI: 1 2022-03-17T19:06:19.2747530Z IS_GHA: 1 2022-03-17T19:06:19.2747749Z IN_WHEEL_TEST: 1 2022-03-17T19:06:19.2748034Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:06:19.2748420Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:06:19.2748736Z PR_LABELS: [] 2022-03-17T19:06:19.2749152Z GITHUB_TOKEN: *** 2022-03-17T19:06:19.2749420Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:06:19.2749678Z PR_NUMBER: 2022-03-17T19:06:19.2749932Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:19.2750221Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:06:19.2750489Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:06:19.2751004Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:06:19.2751557Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:06:19.2751859Z TEST_CONFIG: distributed 2022-03-17T19:06:19.2752100Z SHARD_NUMBER: 1 2022-03-17T19:06:19.2752318Z NUM_TEST_SHARDS: 1 2022-03-17T19:06:19.2752546Z PR_BODY: 2022-03-17T19:06:19.2752777Z AWS_RETRY_MODE: standard 2022-03-17T19:06:19.2753007Z AWS_MAX_ATTEMPTS: 5 2022-03-17T19:06:19.2753235Z ##[endgroup] 2022-03-17T19:06:20.2345374Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2022-03-17T19:06:20.2345835Z Configure a credential helper to remove this warning. See 2022-03-17T19:06:20.2346360Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2022-03-17T19:06:20.2346604Z 2022-03-17T19:06:20.2348796Z Login Succeeded 2022-03-17T19:06:20.2397358Z ##[group]Run retry () { 2022-03-17T19:06:20.2397684Z retry () { 2022-03-17T19:06:20.2397973Z  "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") 2022-03-17T19:06:20.2398215Z } 2022-03-17T19:06:20.2398488Z retry docker pull "${ALPINE_IMAGE}" 2022-03-17T19:06:20.2398830Z # Ensure the working directory gets chowned back to the current user 2022-03-17T19:06:20.2399285Z docker run --pull=never --rm -v "$(pwd)":/v -w /v "${ALPINE_IMAGE}" chown -R "$(id -u):$(id -g)" . 2022-03-17T19:06:20.2412192Z shell: /usr/bin/bash -e {0} 2022-03-17T19:06:20.2412447Z env: 2022-03-17T19:06:20.2412736Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:06:20.2413242Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:06:20.2413734Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:06:20.2414175Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:06:20.2414527Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:06:20.2414773Z IN_CI: 1 2022-03-17T19:06:20.2414994Z IS_GHA: 1 2022-03-17T19:06:20.2415206Z IN_WHEEL_TEST: 1 2022-03-17T19:06:20.2415512Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:06:20.2415882Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:06:20.2416205Z PR_LABELS: [] 2022-03-17T19:06:20.2416719Z GITHUB_TOKEN: *** 2022-03-17T19:06:20.2416978Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:06:20.2417238Z PR_NUMBER: 2022-03-17T19:06:20.2417491Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:20.2417782Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:06:20.2418059Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:06:20.2418545Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:06:20.2419236Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:06:20.2419543Z TEST_CONFIG: distributed 2022-03-17T19:06:20.2419792Z SHARD_NUMBER: 1 2022-03-17T19:06:20.2420100Z NUM_TEST_SHARDS: 1 2022-03-17T19:06:20.2420630Z PR_BODY: 2022-03-17T19:06:20.2420864Z ##[endgroup] 2022-03-17T19:06:20.2802606Z Using default tag: latest 2022-03-17T19:06:20.4676773Z latest: Pulling from tool/alpine 2022-03-17T19:06:20.4677504Z 540db60ca938: Pulling fs layer 2022-03-17T19:06:20.5897341Z 540db60ca938: Verifying Checksum 2022-03-17T19:06:20.5897626Z 540db60ca938: Download complete 2022-03-17T19:06:20.7572189Z 540db60ca938: Pull complete 2022-03-17T19:06:20.7656564Z Digest: sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748 2022-03-17T19:06:20.7689755Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine:latest 2022-03-17T19:06:20.7710395Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine:latest 2022-03-17T19:06:27.1621484Z ##[group]Run rm -rf "${GITHUB_WORKSPACE}" 2022-03-17T19:06:27.1621909Z rm -rf "${GITHUB_WORKSPACE}" 2022-03-17T19:06:27.1622188Z mkdir "${GITHUB_WORKSPACE}" 2022-03-17T19:06:27.1634667Z shell: /usr/bin/bash -e {0} 2022-03-17T19:06:27.1634899Z env: 2022-03-17T19:06:27.1635205Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:06:27.1635715Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:06:27.1636194Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:06:27.1636740Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:06:27.1637145Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:06:27.1637441Z IN_CI: 1 2022-03-17T19:06:27.1637665Z IS_GHA: 1 2022-03-17T19:06:27.1637883Z IN_WHEEL_TEST: 1 2022-03-17T19:06:27.1638189Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:06:27.1638602Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:06:27.1638918Z PR_LABELS: [] 2022-03-17T19:06:27.1639941Z GITHUB_TOKEN: *** 2022-03-17T19:06:27.1640243Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:06:27.1640483Z PR_NUMBER: 2022-03-17T19:06:27.1640752Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:27.1641023Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:06:27.1641297Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:06:27.1641804Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:06:27.1642341Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:06:27.1642663Z TEST_CONFIG: distributed 2022-03-17T19:06:27.1642897Z SHARD_NUMBER: 1 2022-03-17T19:06:27.1643134Z NUM_TEST_SHARDS: 1 2022-03-17T19:06:27.1643372Z PR_BODY: 2022-03-17T19:06:27.1643583Z ##[endgroup] 2022-03-17T19:06:27.1766339Z ##[group]Run seemethere/add-github-ssh-key@v1 2022-03-17T19:06:27.1766709Z with: 2022-03-17T19:06:27.1767276Z GITHUB_TOKEN: *** 2022-03-17T19:06:27.1767548Z activate-with-label: true 2022-03-17T19:06:27.1767815Z label: with-ssh 2022-03-17T19:06:27.1768066Z remove-existing-keys: true 2022-03-17T19:06:27.1768329Z env: 2022-03-17T19:06:27.1768633Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:06:27.1769122Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:06:27.1769616Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:06:27.1770019Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:06:27.1770388Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:06:27.1770635Z IN_CI: 1 2022-03-17T19:06:27.1770835Z IS_GHA: 1 2022-03-17T19:06:27.1771065Z IN_WHEEL_TEST: 1 2022-03-17T19:06:27.1771394Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:06:27.1771771Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:06:27.1772261Z PR_LABELS: [] 2022-03-17T19:06:27.1772669Z GITHUB_TOKEN: *** 2022-03-17T19:06:27.1772969Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:06:27.1773179Z PR_NUMBER: 2022-03-17T19:06:27.1773434Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:27.1773817Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:06:27.1774001Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:06:27.1774490Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:06:27.1775042Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:06:27.1775364Z TEST_CONFIG: distributed 2022-03-17T19:06:27.1775601Z SHARD_NUMBER: 1 2022-03-17T19:06:27.1775844Z NUM_TEST_SHARDS: 1 2022-03-17T19:06:27.1776067Z PR_BODY: 2022-03-17T19:06:27.1776311Z ##[endgroup] 2022-03-17T19:06:27.2565694Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2022-03-17T19:06:27.2619401Z ##[group]Run env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}" 2022-03-17T19:06:27.2619801Z env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}" 2022-03-17T19:06:27.2631522Z shell: /usr/bin/bash -e {0} 2022-03-17T19:06:27.2631773Z env: 2022-03-17T19:06:27.2632062Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:06:27.2632570Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:06:27.2633063Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:06:27.2633467Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:06:27.2633827Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:06:27.2634055Z IN_CI: 1 2022-03-17T19:06:27.2634276Z IS_GHA: 1 2022-03-17T19:06:27.2634507Z IN_WHEEL_TEST: 1 2022-03-17T19:06:27.2634793Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:06:27.2635223Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:06:27.2635531Z PR_LABELS: [] 2022-03-17T19:06:27.2636002Z GITHUB_TOKEN: *** 2022-03-17T19:06:27.2636258Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:06:27.2636517Z PR_NUMBER: 2022-03-17T19:06:27.2636788Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:27.2637058Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:06:27.2637332Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:06:27.2637819Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:06:27.2638374Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:06:27.2638697Z TEST_CONFIG: distributed 2022-03-17T19:06:27.2638930Z SHARD_NUMBER: 1 2022-03-17T19:06:27.2639170Z NUM_TEST_SHARDS: 1 2022-03-17T19:06:27.2639388Z PR_BODY: 2022-03-17T19:06:27.2639618Z ##[endgroup] 2022-03-17T19:06:27.2759337Z ##[group]Run zhouzhuojie/checkout@05b13c9a0d21f08f6d5e64a1d5042246d13619d9 2022-03-17T19:06:27.2759691Z with: 2022-03-17T19:06:27.2759938Z ref: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:27.2760213Z fetch-depth: 0 2022-03-17T19:06:27.2760466Z submodules: recursive 2022-03-17T19:06:27.2760719Z repository: pytorch/pytorch 2022-03-17T19:06:27.2761162Z token: *** 2022-03-17T19:06:27.2761391Z ssh-strict: true 2022-03-17T19:06:27.2761660Z persist-credentials: true 2022-03-17T19:06:27.2761925Z clean: true 2022-03-17T19:06:27.2762134Z lfs: false 2022-03-17T19:06:27.2762352Z env: 2022-03-17T19:06:27.2762635Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:06:27.2763137Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:06:27.2763616Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:06:27.2764081Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:06:27.2764573Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:06:27.2764817Z IN_CI: 1 2022-03-17T19:06:27.2765036Z IS_GHA: 1 2022-03-17T19:06:27.2765247Z IN_WHEEL_TEST: 1 2022-03-17T19:06:27.2765553Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:06:27.2765922Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:06:27.2766245Z PR_LABELS: [] 2022-03-17T19:06:27.2766623Z GITHUB_TOKEN: *** 2022-03-17T19:06:27.2766875Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:06:27.2767131Z PR_NUMBER: 2022-03-17T19:06:27.2767383Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:06:27.2767671Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:06:27.2767940Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:06:27.2768426Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:06:27.2768985Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:06:27.2769294Z TEST_CONFIG: distributed 2022-03-17T19:06:27.2769543Z SHARD_NUMBER: 1 2022-03-17T19:06:27.2769779Z NUM_TEST_SHARDS: 1 2022-03-17T19:06:27.2770000Z PR_BODY: 2022-03-17T19:06:27.2770223Z ##[endgroup] 2022-03-17T19:06:27.3917786Z Syncing repository: pytorch/pytorch 2022-03-17T19:06:27.3919738Z ##[group]Getting Git version info 2022-03-17T19:06:27.3920498Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-03-17T19:06:27.3921173Z [command]/usr/bin/git version 2022-03-17T19:06:27.3921516Z git version 2.32.0 2022-03-17T19:06:27.3922494Z ##[endgroup] 2022-03-17T19:06:27.3925061Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2022-03-17T19:06:27.3929080Z ##[group]Initializing the repository 2022-03-17T19:06:27.3933652Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2022-03-17T19:06:27.3963049Z hint: Using 'master' as the name for the initial branch. This default branch name 2022-03-17T19:06:27.3963677Z hint: is subject to change. To configure the initial branch name to use in all 2022-03-17T19:06:27.3964128Z hint: of your new repositories, which will suppress this warning, call: 2022-03-17T19:06:27.3964449Z hint: 2022-03-17T19:06:27.3964931Z hint: git config --global init.defaultBranch 2022-03-17T19:06:27.3965326Z hint: 2022-03-17T19:06:27.3965834Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2022-03-17T19:06:27.3966418Z hint: 'development'. The just-created branch can be renamed via this command: 2022-03-17T19:06:27.3966719Z hint: 2022-03-17T19:06:27.3967175Z hint: git branch -m 2022-03-17T19:06:27.3967663Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2022-03-17T19:06:27.3975233Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2022-03-17T19:06:27.4002624Z ##[endgroup] 2022-03-17T19:06:27.4003389Z ##[group]Disabling automatic garbage collection 2022-03-17T19:06:27.4007328Z [command]/usr/bin/git config --local gc.auto 0 2022-03-17T19:06:27.4033479Z ##[endgroup] 2022-03-17T19:06:27.4070067Z ##[group]Setting up auth 2022-03-17T19:06:27.4070660Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-03-17T19:06:27.4094028Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-03-17T19:06:27.4448369Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-03-17T19:06:27.4476370Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-03-17T19:06:27.4768944Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-03-17T19:06:27.4828218Z ##[endgroup] 2022-03-17T19:06:27.4829028Z ##[group]Fetching the repository 2022-03-17T19:06:27.4835816Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2022-03-17T19:06:30.4713280Z remote: Enumerating objects: 767821 2022-03-17T19:06:30.4713770Z remote: Enumerating objects: 768219, done. 2022-03-17T19:06:30.4714389Z remote: Counting objects: 0% (1/398) 2022-03-17T19:06:30.4761912Z remote: Counting objects: 1% (4/398) 2022-03-17T19:06:30.4762294Z remote: Counting objects: 2% (8/398) 2022-03-17T19:06:30.4762820Z remote: Counting objects: 3% (12/398) 2022-03-17T19:06:30.4763323Z remote: Counting objects: 4% (16/398) 2022-03-17T19:06:30.4763621Z remote: Counting objects: 5% (20/398) 2022-03-17T19:06:30.4764426Z remote: Counting objects: 6% (24/398) 2022-03-17T19:06:30.4765143Z remote: Counting objects: 7% (28/398) 2022-03-17T19:06:30.4765944Z remote: Counting objects: 8% (32/398) 2022-03-17T19:06:30.4766760Z remote: Counting objects: 9% (36/398) 2022-03-17T19:06:30.4767339Z remote: Counting objects: 10% (40/398) 2022-03-17T19:06:30.4767767Z remote: Counting objects: 11% (44/398) 2022-03-17T19:06:30.4768438Z remote: Counting objects: 12% (48/398) 2022-03-17T19:06:30.4768843Z remote: Counting objects: 13% (52/398) 2022-03-17T19:06:30.4769220Z remote: Counting objects: 14% (56/398) 2022-03-17T19:06:30.4769605Z remote: Counting objects: 15% (60/398) 2022-03-17T19:06:30.4769951Z remote: Counting objects: 16% (64/398) 2022-03-17T19:06:30.4770525Z remote: Counting objects: 17% (68/398) 2022-03-17T19:06:30.4771194Z remote: Counting objects: 18% (72/398) 2022-03-17T19:06:30.4771725Z remote: Counting objects: 19% (76/398) 2022-03-17T19:06:30.4772172Z remote: Counting objects: 20% (80/398) 2022-03-17T19:06:30.4772659Z remote: Counting objects: 21% (84/398) 2022-03-17T19:06:30.4773018Z remote: Counting objects: 22% (88/398) 2022-03-17T19:06:30.4773424Z remote: Counting objects: 23% (92/398) 2022-03-17T19:06:30.4773755Z remote: Counting objects: 24% (96/398) 2022-03-17T19:06:30.4774146Z remote: Counting objects: 25% (100/398) 2022-03-17T19:06:30.4774591Z remote: Counting objects: 26% (104/398) 2022-03-17T19:06:30.4774951Z remote: Counting objects: 27% (108/398) 2022-03-17T19:06:30.4775364Z remote: Counting objects: 28% (112/398) 2022-03-17T19:06:30.4775714Z remote: Counting objects: 29% (116/398) 2022-03-17T19:06:30.4776113Z remote: Counting objects: 30% (120/398) 2022-03-17T19:06:30.4776456Z remote: Counting objects: 31% (124/398) 2022-03-17T19:06:30.4776941Z remote: Counting objects: 32% (128/398) 2022-03-17T19:06:30.4777339Z remote: Counting objects: 33% (132/398) 2022-03-17T19:06:30.4777890Z remote: Counting objects: 34% (136/398) 2022-03-17T19:06:30.4778336Z remote: Counting objects: 35% (140/398) 2022-03-17T19:06:30.4778682Z remote: Counting objects: 36% (144/398) 2022-03-17T19:06:30.4779154Z remote: Counting objects: 37% (148/398) 2022-03-17T19:06:30.4779590Z remote: Counting objects: 38% (152/398) 2022-03-17T19:06:30.4779923Z remote: Counting objects: 39% (156/398) 2022-03-17T19:06:30.4780642Z remote: Counting objects: 40% (160/398) 2022-03-17T19:06:30.4780990Z remote: Counting objects: 41% (164/398) 2022-03-17T19:06:30.4781444Z remote: Counting objects: 42% (168/398) 2022-03-17T19:06:30.4781902Z remote: Counting objects: 43% (172/398) 2022-03-17T19:06:30.4782239Z remote: Counting objects: 44% (176/398) 2022-03-17T19:06:30.4782642Z remote: Counting objects: 45% (180/398) 2022-03-17T19:06:30.4782972Z remote: Counting objects: 46% (184/398) 2022-03-17T19:06:30.4783449Z remote: Counting objects: 47% (188/398) 2022-03-17T19:06:30.4784052Z remote: Counting objects: 48% (192/398) 2022-03-17T19:06:30.4784397Z remote: Counting objects: 49% (196/398) 2022-03-17T19:06:30.4784795Z remote: Counting objects: 50% (199/398) 2022-03-17T19:06:30.4785127Z remote: Counting objects: 51% (203/398) 2022-03-17T19:06:30.4785593Z remote: Counting objects: 52% (207/398) 2022-03-17T19:06:30.4785928Z remote: Counting objects: 53% (211/398) 2022-03-17T19:06:30.4786345Z remote: Counting objects: 54% (215/398) 2022-03-17T19:06:30.4786740Z remote: Counting objects: 55% (219/398) 2022-03-17T19:06:30.4787073Z remote: Counting objects: 56% (223/398) 2022-03-17T19:06:30.4787555Z remote: Counting objects: 57% (227/398) 2022-03-17T19:06:30.4787910Z remote: Counting objects: 58% (231/398) 2022-03-17T19:06:30.4788299Z remote: Counting objects: 59% (235/398) 2022-03-17T19:06:30.4788701Z remote: Counting objects: 60% (239/398) 2022-03-17T19:06:30.4789039Z remote: Counting objects: 61% (243/398) 2022-03-17T19:06:30.4789507Z remote: Counting objects: 62% (247/398) 2022-03-17T19:06:30.4789846Z remote: Counting objects: 63% (251/398) 2022-03-17T19:06:30.4790234Z remote: Counting objects: 64% (255/398) 2022-03-17T19:06:30.4790698Z remote: Counting objects: 65% (259/398) 2022-03-17T19:06:30.4791033Z remote: Counting objects: 66% (263/398) 2022-03-17T19:06:30.4791537Z remote: Counting objects: 67% (267/398) 2022-03-17T19:06:30.4791873Z remote: Counting objects: 68% (271/398) 2022-03-17T19:06:30.4792263Z remote: Counting objects: 69% (275/398) 2022-03-17T19:06:30.4792591Z remote: Counting objects: 70% (279/398) 2022-03-17T19:06:30.4792993Z remote: Counting objects: 71% (283/398) 2022-03-17T19:06:30.4793474Z remote: Counting objects: 72% (287/398) 2022-03-17T19:06:30.4793816Z remote: Counting objects: 73% (291/398) 2022-03-17T19:06:30.4794216Z remote: Counting objects: 74% (295/398) 2022-03-17T19:06:30.4794545Z remote: Counting objects: 75% (299/398) 2022-03-17T19:06:30.4794948Z remote: Counting objects: 76% (303/398) 2022-03-17T19:06:30.4795442Z remote: Counting objects: 77% (307/398) 2022-03-17T19:06:30.4795783Z remote: Counting objects: 78% (311/398) 2022-03-17T19:06:30.4796175Z remote: Counting objects: 79% (315/398) 2022-03-17T19:06:30.4796515Z remote: Counting objects: 80% (319/398) 2022-03-17T19:06:30.4796908Z remote: Counting objects: 81% (323/398) 2022-03-17T19:06:30.4797300Z remote: Counting objects: 82% (327/398) 2022-03-17T19:06:30.4797798Z remote: Counting objects: 83% (331/398) 2022-03-17T19:06:30.4798198Z remote: Counting objects: 84% (335/398) 2022-03-17T19:06:30.4798531Z remote: Counting objects: 85% (339/398) 2022-03-17T19:06:30.4799063Z remote: Counting objects: 86% (343/398) 2022-03-17T19:06:30.4799505Z remote: Counting objects: 87% (347/398) 2022-03-17T19:06:30.4799915Z remote: Counting objects: 88% (351/398) 2022-03-17T19:06:30.4800309Z remote: Counting objects: 89% (355/398) 2022-03-17T19:06:30.4800642Z remote: Counting objects: 90% (359/398) 2022-03-17T19:06:30.4801033Z remote: Counting objects: 91% (363/398) 2022-03-17T19:06:30.4801429Z remote: Counting objects: 92% (367/398) 2022-03-17T19:06:30.4801842Z remote: Counting objects: 93% (371/398) 2022-03-17T19:06:30.4802172Z remote: Counting objects: 94% (375/398) 2022-03-17T19:06:30.4802563Z remote: Counting objects: 95% (379/398) 2022-03-17T19:06:30.4802979Z remote: Counting objects: 96% (383/398) 2022-03-17T19:06:30.4803378Z remote: Counting objects: 97% (387/398) 2022-03-17T19:06:30.4803790Z remote: Counting objects: 98% (391/398) 2022-03-17T19:06:30.4804131Z remote: Counting objects: 99% (395/398) 2022-03-17T19:06:30.4804622Z remote: Counting objects: 100% (398/398) 2022-03-17T19:06:30.4805042Z remote: Counting objects: 100% (398/398), done. 2022-03-17T19:06:30.4805475Z remote: Compressing objects: 0% (1/295) 2022-03-17T19:06:30.4805910Z remote: Compressing objects: 1% (3/295) 2022-03-17T19:06:30.4806264Z remote: Compressing objects: 2% (6/295) 2022-03-17T19:06:30.4813587Z remote: Compressing objects: 3% (9/295) 2022-03-17T19:06:30.4918560Z remote: Compressing objects: 4% (12/295) 2022-03-17T19:06:30.4919047Z remote: Compressing objects: 5% (15/295) 2022-03-17T19:06:30.5062655Z remote: Compressing objects: 6% (18/295) 2022-03-17T19:06:30.5174475Z remote: Compressing objects: 7% (21/295) 2022-03-17T19:06:30.5304680Z remote: Compressing objects: 8% (24/295) 2022-03-17T19:06:30.5468998Z remote: Compressing objects: 9% (27/295) 2022-03-17T19:06:30.5469960Z remote: Compressing objects: 10% (30/295) 2022-03-17T19:06:30.5470711Z remote: Compressing objects: 11% (33/295) 2022-03-17T19:06:30.5471153Z remote: Compressing objects: 12% (36/295) 2022-03-17T19:06:30.5471502Z remote: Compressing objects: 13% (39/295) 2022-03-17T19:06:30.5471913Z remote: Compressing objects: 14% (42/295) 2022-03-17T19:06:30.5472339Z remote: Compressing objects: 15% (45/295) 2022-03-17T19:06:30.5472753Z remote: Compressing objects: 16% (48/295) 2022-03-17T19:06:30.5596069Z remote: Compressing objects: 17% (51/295) 2022-03-17T19:06:30.5596556Z remote: Compressing objects: 18% (54/295) 2022-03-17T19:06:30.5597048Z remote: Compressing objects: 19% (57/295) 2022-03-17T19:06:30.5597474Z remote: Compressing objects: 20% (59/295) 2022-03-17T19:06:30.5597873Z remote: Compressing objects: 21% (62/295) 2022-03-17T19:06:30.5598323Z remote: Compressing objects: 22% (65/295) 2022-03-17T19:06:30.5598672Z remote: Compressing objects: 23% (68/295) 2022-03-17T19:06:30.5599140Z remote: Compressing objects: 24% (71/295) 2022-03-17T19:06:30.5599673Z remote: Compressing objects: 25% (74/295) 2022-03-17T19:06:30.5600073Z remote: Compressing objects: 26% (77/295) 2022-03-17T19:06:30.5647769Z remote: Compressing objects: 27% (80/295) 2022-03-17T19:06:30.5648092Z remote: Compressing objects: 28% (83/295) 2022-03-17T19:06:30.5648413Z remote: Compressing objects: 29% (86/295) 2022-03-17T19:06:30.5648712Z remote: Compressing objects: 30% (89/295) 2022-03-17T19:06:30.5649000Z remote: Compressing objects: 31% (92/295) 2022-03-17T19:06:30.5649325Z remote: Compressing objects: 32% (95/295) 2022-03-17T19:06:30.5649675Z remote: Compressing objects: 33% (98/295) 2022-03-17T19:06:30.5650005Z remote: Compressing objects: 34% (101/295) 2022-03-17T19:06:30.5650306Z remote: Compressing objects: 35% (104/295) 2022-03-17T19:06:30.5650756Z remote: Compressing objects: 36% (107/295) 2022-03-17T19:06:30.5651075Z remote: Compressing objects: 37% (110/295) 2022-03-17T19:06:30.5651420Z remote: Compressing objects: 38% (113/295) 2022-03-17T19:06:30.5651667Z remote: Compressing objects: 39% (116/295) 2022-03-17T19:06:30.5651992Z remote: Compressing objects: 40% (118/295) 2022-03-17T19:06:30.5652315Z remote: Compressing objects: 41% (121/295) 2022-03-17T19:06:30.5652611Z remote: Compressing objects: 42% (124/295) 2022-03-17T19:06:30.5652926Z remote: Compressing objects: 43% (127/295) 2022-03-17T19:06:30.5653223Z remote: Compressing objects: 44% (130/295) 2022-03-17T19:06:30.5653562Z remote: Compressing objects: 45% (133/295) 2022-03-17T19:06:30.5653853Z remote: Compressing objects: 46% (136/295) 2022-03-17T19:06:30.5654244Z remote: Compressing objects: 47% (139/295) 2022-03-17T19:06:30.5654560Z remote: Compressing objects: 48% (142/295) 2022-03-17T19:06:30.5654966Z remote: Compressing objects: 49% (145/295) 2022-03-17T19:06:30.5655257Z remote: Compressing objects: 50% (148/295) 2022-03-17T19:06:30.5655571Z remote: Compressing objects: 51% (151/295) 2022-03-17T19:06:30.5655864Z remote: Compressing objects: 52% (154/295) 2022-03-17T19:06:30.5656182Z remote: Compressing objects: 53% (157/295) 2022-03-17T19:06:30.5656476Z remote: Compressing objects: 54% (160/295) 2022-03-17T19:06:30.5656780Z remote: Compressing objects: 55% (163/295) 2022-03-17T19:06:30.5657074Z remote: Compressing objects: 56% (166/295) 2022-03-17T19:06:30.5657433Z remote: Compressing objects: 57% (169/295) 2022-03-17T19:06:30.5657748Z remote: Compressing objects: 58% (172/295) 2022-03-17T19:06:30.5658043Z remote: Compressing objects: 59% (175/295) 2022-03-17T19:06:30.5658356Z remote: Compressing objects: 60% (177/295) 2022-03-17T19:06:30.5658679Z remote: Compressing objects: 61% (180/295) 2022-03-17T19:06:30.5658971Z remote: Compressing objects: 62% (183/295) 2022-03-17T19:06:30.5659284Z remote: Compressing objects: 63% (186/295) 2022-03-17T19:06:30.5659580Z remote: Compressing objects: 64% (189/295) 2022-03-17T19:06:30.5659893Z remote: Compressing objects: 65% (192/295) 2022-03-17T19:06:30.5660184Z remote: Compressing objects: 66% (195/295) 2022-03-17T19:06:30.5660741Z remote: Compressing objects: 67% (198/295) 2022-03-17T19:06:30.5661060Z remote: Compressing objects: 68% (201/295) 2022-03-17T19:06:30.5661357Z remote: Compressing objects: 69% (204/295) 2022-03-17T19:06:30.5661666Z remote: Compressing objects: 70% (207/295) 2022-03-17T19:06:30.5661960Z remote: Compressing objects: 71% (210/295) 2022-03-17T19:06:30.5662279Z remote: Compressing objects: 72% (213/295) 2022-03-17T19:06:30.5662593Z remote: Compressing objects: 73% (216/295) 2022-03-17T19:06:30.5662888Z remote: Compressing objects: 74% (219/295) 2022-03-17T19:06:30.5663208Z remote: Compressing objects: 75% (222/295) 2022-03-17T19:06:30.5663505Z remote: Compressing objects: 76% (225/295) 2022-03-17T19:06:30.5663816Z remote: Compressing objects: 77% (228/295) 2022-03-17T19:06:30.5664126Z remote: Compressing objects: 78% (231/295) 2022-03-17T19:06:30.5664424Z remote: Compressing objects: 79% (234/295) 2022-03-17T19:06:30.5664781Z remote: Compressing objects: 80% (236/295) 2022-03-17T19:06:30.5665033Z remote: Compressing objects: 81% (239/295) 2022-03-17T19:06:30.5665342Z remote: Compressing objects: 82% (242/295) 2022-03-17T19:06:30.5665640Z remote: Compressing objects: 83% (245/295) 2022-03-17T19:06:30.5665961Z remote: Compressing objects: 84% (248/295) 2022-03-17T19:06:30.5666267Z remote: Compressing objects: 85% (251/295) 2022-03-17T19:06:30.5666668Z remote: Compressing objects: 86% (254/295) 2022-03-17T19:06:30.5667020Z remote: Compressing objects: 87% (257/295) 2022-03-17T19:06:30.5667304Z remote: Compressing objects: 88% (260/295) 2022-03-17T19:06:30.5667609Z remote: Compressing objects: 89% (263/295) 2022-03-17T19:06:30.5667956Z remote: Compressing objects: 90% (266/295) 2022-03-17T19:06:30.5668219Z remote: Compressing objects: 91% (269/295) 2022-03-17T19:06:30.5668592Z remote: Compressing objects: 92% (272/295) 2022-03-17T19:06:30.5668884Z remote: Compressing objects: 93% (275/295) 2022-03-17T19:06:30.5669199Z remote: Compressing objects: 94% (278/295) 2022-03-17T19:06:30.5669515Z remote: Compressing objects: 95% (281/295) 2022-03-17T19:06:30.5669824Z remote: Compressing objects: 96% (284/295) 2022-03-17T19:06:30.5670118Z remote: Compressing objects: 97% (287/295) 2022-03-17T19:06:30.5670415Z remote: Compressing objects: 98% (290/295) 2022-03-17T19:06:30.5670737Z remote: Compressing objects: 99% (293/295) 2022-03-17T19:06:30.5671131Z remote: Compressing objects: 100% (295/295) 2022-03-17T19:06:30.5671460Z remote: Compressing objects: 100% (295/295), done. 2022-03-17T19:06:30.6451906Z Receiving objects: 0% (1/768219) 2022-03-17T19:06:30.7269814Z Receiving objects: 1% (7683/768219) 2022-03-17T19:06:30.8250103Z Receiving objects: 2% (15365/768219) 2022-03-17T19:06:30.9253444Z Receiving objects: 3% (23047/768219) 2022-03-17T19:06:31.0253557Z Receiving objects: 4% (30729/768219) 2022-03-17T19:06:31.1309311Z Receiving objects: 5% (38411/768219) 2022-03-17T19:06:31.2321686Z Receiving objects: 6% (46094/768219), 12.87 MiB | 25.78 MiB/s 2022-03-17T19:06:31.3315159Z Receiving objects: 7% (53776/768219), 12.87 MiB | 25.78 MiB/s 2022-03-17T19:06:31.4288200Z Receiving objects: 8% (61458/768219), 12.87 MiB | 25.78 MiB/s 2022-03-17T19:06:31.5338190Z Receiving objects: 9% (69140/768219), 12.87 MiB | 25.78 MiB/s 2022-03-17T19:06:31.5618242Z Receiving objects: 10% (76822/768219), 12.87 MiB | 25.78 MiB/s 2022-03-17T19:06:31.6561269Z Receiving objects: 10% (78576/768219), 12.87 MiB | 25.78 MiB/s 2022-03-17T19:06:31.7452047Z Receiving objects: 11% (84505/768219), 25.86 MiB | 25.88 MiB/s 2022-03-17T19:06:32.0335735Z Receiving objects: 12% (92187/768219), 25.86 MiB | 25.88 MiB/s 2022-03-17T19:06:32.3630790Z Receiving objects: 13% (99869/768219), 25.86 MiB | 25.88 MiB/s 2022-03-17T19:06:32.5619276Z Receiving objects: 14% (107551/768219), 38.03 MiB | 25.37 MiB/s 2022-03-17T19:06:32.8230691Z Receiving objects: 14% (110042/768219), 38.03 MiB | 25.37 MiB/s 2022-03-17T19:06:33.0381407Z Receiving objects: 15% (115233/768219), 52.35 MiB | 26.18 MiB/s 2022-03-17T19:06:33.3047156Z Receiving objects: 16% (122916/768219), 52.35 MiB | 26.18 MiB/s 2022-03-17T19:06:33.5618448Z Receiving objects: 17% (130598/768219), 66.23 MiB | 26.51 MiB/s 2022-03-17T19:06:33.6422810Z Receiving objects: 17% (136416/768219), 66.23 MiB | 26.51 MiB/s 2022-03-17T19:06:33.9213422Z Receiving objects: 18% (138280/768219), 85.54 MiB | 28.53 MiB/s 2022-03-17T19:06:34.1552734Z Receiving objects: 19% (145962/768219), 85.54 MiB | 28.53 MiB/s 2022-03-17T19:06:34.4115417Z Receiving objects: 20% (153644/768219), 105.30 MiB | 30.10 MiB/s 2022-03-17T19:06:34.5618928Z Receiving objects: 21% (161326/768219), 105.30 MiB | 30.10 MiB/s 2022-03-17T19:06:34.6592828Z Receiving objects: 21% (166002/768219), 105.30 MiB | 30.10 MiB/s 2022-03-17T19:06:34.8758509Z Receiving objects: 22% (169009/768219), 124.90 MiB | 31.24 MiB/s 2022-03-17T19:06:35.0701618Z Receiving objects: 23% (176691/768219), 124.90 MiB | 31.24 MiB/s 2022-03-17T19:06:35.2146683Z Receiving objects: 24% (184373/768219), 143.70 MiB | 31.95 MiB/s 2022-03-17T19:06:35.3715320Z Receiving objects: 25% (192055/768219), 143.70 MiB | 31.95 MiB/s 2022-03-17T19:06:35.5406656Z Receiving objects: 26% (199737/768219), 143.70 MiB | 31.95 MiB/s 2022-03-17T19:06:35.5622717Z Receiving objects: 27% (207420/768219), 143.70 MiB | 31.95 MiB/s 2022-03-17T19:06:35.7406858Z Receiving objects: 27% (208057/768219), 162.87 MiB | 33.34 MiB/s 2022-03-17T19:06:35.9325109Z Receiving objects: 28% (215102/768219), 162.87 MiB | 33.34 MiB/s 2022-03-17T19:06:36.1096851Z Receiving objects: 29% (222784/768219), 162.87 MiB | 33.34 MiB/s 2022-03-17T19:06:36.3335695Z Receiving objects: 30% (230466/768219), 181.98 MiB | 34.70 MiB/s 2022-03-17T19:06:36.5205773Z Receiving objects: 31% (238148/768219), 181.98 MiB | 34.70 MiB/s 2022-03-17T19:06:36.5623663Z Receiving objects: 32% (245831/768219), 181.98 MiB | 34.70 MiB/s 2022-03-17T19:06:36.6652709Z Receiving objects: 32% (248299/768219), 201.12 MiB | 36.25 MiB/s 2022-03-17T19:06:36.7844952Z Receiving objects: 33% (253513/768219), 201.12 MiB | 36.25 MiB/s 2022-03-17T19:06:36.9790088Z Receiving objects: 34% (261195/768219), 201.12 MiB | 36.25 MiB/s 2022-03-17T19:06:37.2543901Z Receiving objects: 35% (268877/768219), 201.12 MiB | 36.25 MiB/s 2022-03-17T19:06:37.4078677Z Receiving objects: 36% (276559/768219), 219.87 MiB | 37.23 MiB/s 2022-03-17T19:06:37.5613956Z Receiving objects: 37% (284242/768219), 219.87 MiB | 37.23 MiB/s 2022-03-17T19:06:37.5624029Z Receiving objects: 38% (291924/768219), 219.87 MiB | 37.23 MiB/s 2022-03-17T19:06:37.7356382Z Receiving objects: 38% (291955/768219), 219.87 MiB | 37.23 MiB/s 2022-03-17T19:06:37.8631957Z Receiving objects: 39% (299606/768219), 236.82 MiB | 37.91 MiB/s 2022-03-17T19:06:38.0318660Z Receiving objects: 40% (307288/768219), 236.82 MiB | 37.91 MiB/s 2022-03-17T19:06:38.1894214Z Receiving objects: 41% (314970/768219), 236.82 MiB | 37.91 MiB/s 2022-03-17T19:06:38.3207921Z Receiving objects: 42% (322652/768219), 256.37 MiB | 37.96 MiB/s 2022-03-17T19:06:38.4614216Z Receiving objects: 43% (330335/768219), 256.37 MiB | 37.96 MiB/s 2022-03-17T19:06:38.5619668Z Receiving objects: 44% (338017/768219), 256.37 MiB | 37.96 MiB/s 2022-03-17T19:06:38.6827830Z Receiving objects: 44% (340706/768219), 256.37 MiB | 37.96 MiB/s 2022-03-17T19:06:38.8322391Z Receiving objects: 45% (345699/768219), 275.17 MiB | 37.75 MiB/s 2022-03-17T19:06:38.9587827Z Receiving objects: 46% (353381/768219), 275.17 MiB | 37.75 MiB/s 2022-03-17T19:06:39.0995599Z Receiving objects: 47% (361063/768219), 275.17 MiB | 37.75 MiB/s 2022-03-17T19:06:39.2184426Z Receiving objects: 48% (368746/768219), 294.98 MiB | 37.80 MiB/s 2022-03-17T19:06:39.3597771Z Receiving objects: 49% (376428/768219), 294.98 MiB | 37.80 MiB/s 2022-03-17T19:06:39.4368459Z Receiving objects: 50% (384110/768219), 294.98 MiB | 37.80 MiB/s 2022-03-17T19:06:39.5116733Z Receiving objects: 51% (391792/768219), 294.98 MiB | 37.80 MiB/s 2022-03-17T19:06:39.5627539Z Receiving objects: 52% (399474/768219), 294.98 MiB | 37.80 MiB/s 2022-03-17T19:06:40.5629235Z Receiving objects: 52% (400252/768219), 315.16 MiB | 38.10 MiB/s 2022-03-17T19:06:40.7843578Z Receiving objects: 52% (406362/768219), 362.55 MiB | 40.13 MiB/s 2022-03-17T19:06:41.5506877Z Receiving objects: 53% (407157/768219), 362.55 MiB | 40.13 MiB/s 2022-03-17T19:06:41.5619198Z Receiving objects: 54% (414839/768219), 388.69 MiB | 41.68 MiB/s 2022-03-17T19:06:41.6601092Z Receiving objects: 54% (415663/768219), 388.69 MiB | 41.68 MiB/s 2022-03-17T19:06:41.7902993Z Receiving objects: 55% (422521/768219), 420.89 MiB | 44.67 MiB/s 2022-03-17T19:06:41.8896078Z Receiving objects: 56% (430203/768219), 420.89 MiB | 44.67 MiB/s 2022-03-17T19:06:42.5630679Z Receiving objects: 57% (437885/768219), 420.89 MiB | 44.67 MiB/s 2022-03-17T19:06:43.5631418Z Receiving objects: 57% (442954/768219), 465.22 MiB | 46.41 MiB/s 2022-03-17T19:06:43.9368191Z Receiving objects: 57% (442955/768219), 512.77 MiB | 48.40 MiB/s 2022-03-17T19:06:44.0417136Z Receiving objects: 58% (445568/768219), 512.77 MiB | 48.40 MiB/s 2022-03-17T19:06:44.0918511Z Receiving objects: 59% (453250/768219), 512.77 MiB | 48.40 MiB/s 2022-03-17T19:06:44.1908917Z Receiving objects: 60% (460932/768219), 535.23 MiB | 48.90 MiB/s 2022-03-17T19:06:44.4360085Z Receiving objects: 61% (468614/768219), 535.23 MiB | 48.90 MiB/s 2022-03-17T19:06:44.5099290Z Receiving objects: 62% (476296/768219), 535.23 MiB | 48.90 MiB/s 2022-03-17T19:06:44.5488754Z Receiving objects: 63% (483978/768219), 535.23 MiB | 48.90 MiB/s 2022-03-17T19:06:44.5620088Z Receiving objects: 64% (491661/768219), 535.23 MiB | 48.90 MiB/s 2022-03-17T19:06:44.6225975Z Receiving objects: 64% (493200/768219), 535.23 MiB | 48.90 MiB/s 2022-03-17T19:06:44.6717029Z Receiving objects: 65% (499343/768219), 556.75 MiB | 48.16 MiB/s 2022-03-17T19:06:44.7528242Z Receiving objects: 66% (507025/768219), 556.75 MiB | 48.16 MiB/s 2022-03-17T19:06:44.8063851Z Receiving objects: 67% (514707/768219), 556.75 MiB | 48.16 MiB/s 2022-03-17T19:06:44.8551272Z Receiving objects: 68% (522389/768219), 556.75 MiB | 48.16 MiB/s 2022-03-17T19:06:44.9385910Z Receiving objects: 69% (530072/768219), 556.75 MiB | 48.16 MiB/s 2022-03-17T19:06:44.9835121Z Receiving objects: 70% (537754/768219), 556.75 MiB | 48.16 MiB/s 2022-03-17T19:06:45.1653249Z Receiving objects: 71% (545436/768219), 556.75 MiB | 48.16 MiB/s 2022-03-17T19:06:45.2607753Z Receiving objects: 72% (553118/768219), 576.36 MiB | 47.51 MiB/s 2022-03-17T19:06:45.3307000Z Receiving objects: 73% (560800/768219), 576.36 MiB | 47.51 MiB/s 2022-03-17T19:06:45.3988166Z Receiving objects: 74% (568483/768219), 576.36 MiB | 47.51 MiB/s 2022-03-17T19:06:45.4600484Z Receiving objects: 75% (576165/768219), 576.36 MiB | 47.51 MiB/s 2022-03-17T19:06:45.5211683Z Receiving objects: 76% (583847/768219), 576.36 MiB | 47.51 MiB/s 2022-03-17T19:06:45.5618284Z Receiving objects: 77% (591529/768219), 576.36 MiB | 47.51 MiB/s 2022-03-17T19:06:45.5719110Z Receiving objects: 77% (598502/768219), 576.36 MiB | 47.51 MiB/s 2022-03-17T19:06:45.7374635Z Receiving objects: 78% (599211/768219), 596.30 MiB | 46.13 MiB/s 2022-03-17T19:06:45.8619974Z Receiving objects: 79% (606894/768219), 596.30 MiB | 46.13 MiB/s 2022-03-17T19:06:45.9136354Z Receiving objects: 80% (614576/768219), 596.30 MiB | 46.13 MiB/s 2022-03-17T19:06:46.0194780Z Receiving objects: 81% (622258/768219), 596.30 MiB | 46.13 MiB/s 2022-03-17T19:06:46.0745055Z Receiving objects: 82% (629940/768219), 596.30 MiB | 46.13 MiB/s 2022-03-17T19:06:46.1561201Z Receiving objects: 83% (637622/768219), 612.76 MiB | 42.64 MiB/s 2022-03-17T19:06:46.5634003Z Receiving objects: 84% (645304/768219), 612.76 MiB | 42.64 MiB/s 2022-03-17T19:06:47.2002679Z Receiving objects: 84% (649972/768219), 635.74 MiB | 43.16 MiB/s 2022-03-17T19:06:47.2951360Z Receiving objects: 85% (652987/768219), 659.40 MiB | 43.15 MiB/s 2022-03-17T19:06:47.4080840Z Receiving objects: 86% (660669/768219), 659.40 MiB | 43.15 MiB/s 2022-03-17T19:06:47.5618293Z Receiving objects: 87% (668351/768219), 659.40 MiB | 43.15 MiB/s 2022-03-17T19:06:47.5864118Z Receiving objects: 87% (675049/768219), 659.40 MiB | 43.15 MiB/s 2022-03-17T19:06:47.7329386Z Receiving objects: 88% (676033/768219), 680.80 MiB | 42.64 MiB/s 2022-03-17T19:06:47.8773466Z Receiving objects: 89% (683715/768219), 680.80 MiB | 42.64 MiB/s 2022-03-17T19:06:47.9900572Z Receiving objects: 90% (691398/768219), 680.80 MiB | 42.64 MiB/s 2022-03-17T19:06:48.1057457Z Receiving objects: 91% (699080/768219), 680.80 MiB | 42.64 MiB/s 2022-03-17T19:06:48.1855854Z Receiving objects: 92% (706762/768219), 701.54 MiB | 41.95 MiB/s 2022-03-17T19:06:48.2712736Z Receiving objects: 93% (714444/768219), 701.54 MiB | 41.95 MiB/s 2022-03-17T19:06:48.3462785Z Receiving objects: 94% (722126/768219), 701.54 MiB | 41.95 MiB/s 2022-03-17T19:06:48.4952462Z Receiving objects: 95% (729809/768219), 701.54 MiB | 41.95 MiB/s 2022-03-17T19:06:48.5575777Z Receiving objects: 96% (737491/768219), 701.54 MiB | 41.95 MiB/s 2022-03-17T19:06:48.5618019Z Receiving objects: 97% (745173/768219), 701.54 MiB | 41.95 MiB/s 2022-03-17T19:06:48.6096406Z Receiving objects: 97% (745792/768219), 701.54 MiB | 41.95 MiB/s 2022-03-17T19:06:48.6974146Z Receiving objects: 98% (752855/768219), 718.80 MiB | 40.79 MiB/s 2022-03-17T19:06:48.7861536Z Receiving objects: 99% (760537/768219), 718.80 MiB | 40.79 MiB/s 2022-03-17T19:06:48.7862289Z remote: Total 768219 (delta 245), reused 199 (delta 103), pack-reused 767821 2022-03-17T19:06:48.7888522Z Receiving objects: 100% (768219/768219), 718.80 MiB | 40.79 MiB/s 2022-03-17T19:06:48.7888859Z Receiving objects: 100% (768219/768219), 727.11 MiB | 39.89 MiB/s, done. 2022-03-17T19:06:48.9022316Z Resolving deltas: 0% (0/622466) 2022-03-17T19:06:48.9204332Z Resolving deltas: 1% (6225/622466) 2022-03-17T19:06:48.9400320Z Resolving deltas: 2% (12450/622466) 2022-03-17T19:06:48.9563696Z Resolving deltas: 3% (18674/622466) 2022-03-17T19:06:49.0071587Z Resolving deltas: 4% (24900/622466) 2022-03-17T19:06:49.0381365Z Resolving deltas: 5% (31124/622466) 2022-03-17T19:06:49.1066224Z Resolving deltas: 6% (37348/622466) 2022-03-17T19:06:49.2212124Z Resolving deltas: 7% (43573/622466) 2022-03-17T19:06:49.2501666Z Resolving deltas: 8% (49798/622466) 2022-03-17T19:06:49.2819719Z Resolving deltas: 9% (56022/622466) 2022-03-17T19:06:49.3094171Z Resolving deltas: 10% (62247/622466) 2022-03-17T19:06:49.3454670Z Resolving deltas: 11% (68472/622466) 2022-03-17T19:06:49.3816109Z Resolving deltas: 12% (74696/622466) 2022-03-17T19:06:49.4656862Z Resolving deltas: 13% (80922/622466) 2022-03-17T19:06:49.8874842Z Resolving deltas: 14% (87146/622466) 2022-03-17T19:06:50.1683502Z Resolving deltas: 14% (91022/622466) 2022-03-17T19:06:50.2122084Z Resolving deltas: 15% (93371/622466) 2022-03-17T19:06:50.2658484Z Resolving deltas: 16% (99595/622466) 2022-03-17T19:06:50.3160470Z Resolving deltas: 17% (105822/622466) 2022-03-17T19:06:50.3657798Z Resolving deltas: 18% (112044/622466) 2022-03-17T19:06:50.4092556Z Resolving deltas: 19% (118269/622466) 2022-03-17T19:06:50.5616178Z Resolving deltas: 20% (124494/622466) 2022-03-17T19:06:50.6007778Z Resolving deltas: 21% (130718/622466) 2022-03-17T19:06:50.6422982Z Resolving deltas: 22% (136943/622466) 2022-03-17T19:06:50.8083679Z Resolving deltas: 23% (143168/622466) 2022-03-17T19:06:50.8331372Z Resolving deltas: 24% (149393/622466) 2022-03-17T19:06:50.8874762Z Resolving deltas: 25% (155618/622466) 2022-03-17T19:06:50.9389725Z Resolving deltas: 25% (160154/622466) 2022-03-17T19:06:50.9702192Z Resolving deltas: 26% (161842/622466) 2022-03-17T19:06:51.0504052Z Resolving deltas: 27% (168067/622466) 2022-03-17T19:06:51.1873292Z Resolving deltas: 28% (174291/622466) 2022-03-17T19:06:51.2172926Z Resolving deltas: 29% (180516/622466) 2022-03-17T19:06:51.2278474Z Resolving deltas: 30% (186740/622466) 2022-03-17T19:06:51.2403478Z Resolving deltas: 31% (192965/622466) 2022-03-17T19:06:51.2495380Z Resolving deltas: 32% (199191/622466) 2022-03-17T19:06:51.2732101Z Resolving deltas: 33% (205414/622466) 2022-03-17T19:06:51.2830589Z Resolving deltas: 34% (211639/622466) 2022-03-17T19:06:51.2962286Z Resolving deltas: 35% (217865/622466) 2022-03-17T19:06:51.3083251Z Resolving deltas: 36% (224089/622466) 2022-03-17T19:06:51.3170332Z Resolving deltas: 37% (230313/622466) 2022-03-17T19:06:51.3257099Z Resolving deltas: 38% (236539/622466) 2022-03-17T19:06:51.3471154Z Resolving deltas: 39% (242764/622466) 2022-03-17T19:06:51.3591255Z Resolving deltas: 40% (248987/622466) 2022-03-17T19:06:51.3811247Z Resolving deltas: 41% (255212/622466) 2022-03-17T19:06:51.4165604Z Resolving deltas: 42% (261436/622466) 2022-03-17T19:06:51.4328829Z Resolving deltas: 43% (267661/622466) 2022-03-17T19:06:51.4537037Z Resolving deltas: 44% (273886/622466) 2022-03-17T19:06:51.4729328Z Resolving deltas: 45% (280110/622466) 2022-03-17T19:06:51.4955561Z Resolving deltas: 46% (286335/622466) 2022-03-17T19:06:51.5378286Z Resolving deltas: 47% (292560/622466) 2022-03-17T19:06:51.5494252Z Resolving deltas: 48% (298784/622466) 2022-03-17T19:06:51.5632122Z Resolving deltas: 49% (305009/622466) 2022-03-17T19:06:51.5914948Z Resolving deltas: 50% (311233/622466) 2022-03-17T19:06:51.6048163Z Resolving deltas: 51% (317459/622466) 2022-03-17T19:06:51.6913155Z Resolving deltas: 52% (323683/622466) 2022-03-17T19:06:51.7171061Z Resolving deltas: 53% (329907/622466) 2022-03-17T19:06:51.7517679Z Resolving deltas: 54% (336133/622466) 2022-03-17T19:06:51.7627895Z Resolving deltas: 55% (342357/622466) 2022-03-17T19:06:51.7820808Z Resolving deltas: 56% (348581/622466) 2022-03-17T19:06:51.7975887Z Resolving deltas: 57% (354806/622466) 2022-03-17T19:06:51.8291864Z Resolving deltas: 58% (361032/622466) 2022-03-17T19:06:51.8409305Z Resolving deltas: 59% (367255/622466) 2022-03-17T19:06:51.8554328Z Resolving deltas: 60% (373480/622466) 2022-03-17T19:06:51.8872799Z Resolving deltas: 61% (379709/622466) 2022-03-17T19:06:51.8889611Z Resolving deltas: 61% (385371/622466) 2022-03-17T19:06:52.0777901Z Resolving deltas: 62% (385929/622466) 2022-03-17T19:06:52.0914734Z Resolving deltas: 63% (392154/622466) 2022-03-17T19:06:52.1063471Z Resolving deltas: 64% (398380/622466) 2022-03-17T19:06:52.1196988Z Resolving deltas: 65% (404603/622466) 2022-03-17T19:06:52.1307460Z Resolving deltas: 66% (410828/622466) 2022-03-17T19:06:52.1607194Z Resolving deltas: 67% (417055/622466) 2022-03-17T19:06:52.1713644Z Resolving deltas: 68% (423277/622466) 2022-03-17T19:06:52.1971471Z Resolving deltas: 69% (429503/622466) 2022-03-17T19:06:52.2110066Z Resolving deltas: 70% (435727/622466) 2022-03-17T19:06:52.2199995Z Resolving deltas: 71% (441952/622466) 2022-03-17T19:06:52.2516371Z Resolving deltas: 72% (448176/622466) 2022-03-17T19:06:52.3499323Z Resolving deltas: 73% (454401/622466) 2022-03-17T19:06:52.3697190Z Resolving deltas: 74% (460625/622466) 2022-03-17T19:06:52.5315910Z Resolving deltas: 75% (466851/622466) 2022-03-17T19:06:52.5588216Z Resolving deltas: 76% (473075/622466) 2022-03-17T19:06:52.6009822Z Resolving deltas: 77% (479299/622466) 2022-03-17T19:06:52.6192580Z Resolving deltas: 78% (485524/622466) 2022-03-17T19:06:52.6348439Z Resolving deltas: 79% (491749/622466) 2022-03-17T19:06:52.6689347Z Resolving deltas: 80% (497973/622466) 2022-03-17T19:06:52.6849038Z Resolving deltas: 81% (504198/622466) 2022-03-17T19:06:52.7104202Z Resolving deltas: 82% (510424/622466) 2022-03-17T19:06:52.7335079Z Resolving deltas: 83% (516647/622466) 2022-03-17T19:06:52.7589741Z Resolving deltas: 84% (522872/622466) 2022-03-17T19:06:52.7756162Z Resolving deltas: 85% (529097/622466) 2022-03-17T19:06:52.7914615Z Resolving deltas: 86% (535321/622466) 2022-03-17T19:06:52.8078652Z Resolving deltas: 87% (541546/622466) 2022-03-17T19:06:52.8326622Z Resolving deltas: 88% (547771/622466) 2022-03-17T19:06:52.8607465Z Resolving deltas: 89% (553995/622466) 2022-03-17T19:06:52.8774397Z Resolving deltas: 90% (560224/622466) 2022-03-17T19:06:52.8872521Z Resolving deltas: 91% (566446/622466) 2022-03-17T19:06:52.9055624Z Resolving deltas: 91% (569849/622466) 2022-03-17T19:06:52.9214049Z Resolving deltas: 92% (572670/622466) 2022-03-17T19:06:52.9331112Z Resolving deltas: 93% (578894/622466) 2022-03-17T19:06:52.9434514Z Resolving deltas: 94% (585121/622466) 2022-03-17T19:06:52.9549598Z Resolving deltas: 95% (591344/622466) 2022-03-17T19:06:52.9688139Z Resolving deltas: 96% (597568/622466) 2022-03-17T19:06:52.9798697Z Resolving deltas: 97% (603793/622466) 2022-03-17T19:06:52.9897804Z Resolving deltas: 98% (610017/622466) 2022-03-17T19:06:53.1198068Z Resolving deltas: 99% (616243/622466) 2022-03-17T19:06:53.1198455Z Resolving deltas: 100% (622466/622466) 2022-03-17T19:06:53.1198750Z Resolving deltas: 100% (622466/622466), done. 2022-03-17T19:07:02.7164795Z From https://github.com/pytorch/pytorch 2022-03-17T19:07:02.7165392Z * [new branch] 1.6 -> origin/1.6 2022-03-17T19:07:02.7165858Z * [new branch] 1.6_doc_references -> origin/1.6_doc_references 2022-03-17T19:07:02.7167187Z * [new branch] 1.7 -> origin/1.7 2022-03-17T19:07:02.7168963Z * [new branch] 20_12_3_devel -> origin/20_12_3_devel 2022-03-17T19:07:02.7170224Z * [new branch] 21250-2 -> origin/21250-2 2022-03-17T19:07:02.7171974Z * [new branch] 32495checkout -> origin/32495checkout 2022-03-17T19:07:02.7173864Z * [new branch] 42835-cherry-pick -> origin/42835-cherry-pick 2022-03-17T19:07:02.7175153Z * [new branch] 53867-memory-planninger-output-tensor -> origin/53867-memory-planninger-output-tensor 2022-03-17T19:07:02.7176503Z * [new branch] 71806_migrate_default_branch_to_main -> origin/71806_migrate_default_branch_to_main 2022-03-17T19:07:02.7178706Z * [new branch] 71806_migrate_master_branch_to_main -> origin/71806_migrate_master_branch_to_main 2022-03-17T19:07:02.7181258Z * [new branch] 71806_support_main_branch_in_github_workflows -> origin/71806_support_main_branch_in_github_workflows 2022-03-17T19:07:02.7197415Z * [new branch] Add-new-process-group-_internal_ucc -> origin/Add-new-process-group-_internal_ucc 2022-03-17T19:07:02.7198796Z * [new branch] ArchiveStream -> origin/ArchiveStream 2022-03-17T19:07:02.7200815Z * [new branch] BUILD_SPLIT_CUDA_AUTO_DETECT_CPP_EXT -> origin/BUILD_SPLIT_CUDA_AUTO_DETECT_CPP_EXT 2022-03-17T19:07:02.7202095Z * [new branch] CUDA_TEST -> origin/CUDA_TEST 2022-03-17T19:07:02.7204010Z * [new branch] Chillee-patch-1 -> origin/Chillee-patch-1 2022-03-17T19:07:02.7205270Z * [new branch] CodeGenRefactor -> origin/CodeGenRefactor 2022-03-17T19:07:02.7206868Z * [new branch] CodeGenRefactor_diff -> origin/CodeGenRefactor_diff 2022-03-17T19:07:02.7208171Z * [new branch] DIFF_B_1 -> origin/DIFF_B_1 2022-03-17T19:07:02.7209904Z * [new branch] DIFF_B_2 -> origin/DIFF_B_2 2022-03-17T19:07:02.7211160Z * [new branch] DataPipeForkDemux -> origin/DataPipeForkDemux 2022-03-17T19:07:02.7212906Z * [new branch] DataPipesAPI -> origin/DataPipesAPI 2022-03-17T19:07:02.7214153Z * [new branch] DataPipesAPIClarity -> origin/DataPipesAPIClarity 2022-03-17T19:07:02.7215757Z * [new branch] DataPipes_Simple_Unbatch -> origin/DataPipes_Simple_Unbatch 2022-03-17T19:07:02.7217569Z * [new branch] FE2774773A77F1D7EA51AFAD32EA53C9 -> origin/FE2774773A77F1D7EA51AFAD32EA53C9 2022-03-17T19:07:02.7218778Z * [new branch] FOO1 -> origin/FOO1 2022-03-17T19:07:02.7220342Z * [new branch] Fix-master-due-to-62520 -> origin/Fix-master-due-to-62520 2022-03-17T19:07:02.7222354Z * [new branch] FixCombiningWarning -> origin/FixCombiningWarning 2022-03-17T19:07:02.7223752Z * [new branch] Improve_SetRNGState_Doc -> origin/Improve_SetRNGState_Doc 2022-03-17T19:07:02.7225568Z * [new branch] IvanYashchuk-patch-1 -> origin/IvanYashchuk-patch-1 2022-03-17T19:07:02.7226886Z * [new branch] IvanYashchuk-patch-2 -> origin/IvanYashchuk-patch-2 2022-03-17T19:07:02.7228452Z * [new branch] MTH_DDP -> origin/MTH_DDP 2022-03-17T19:07:02.7229826Z * [new branch] RemoveTensorOptionsRC -> origin/RemoveTensorOptionsRC 2022-03-17T19:07:02.7231466Z * [new branch] RemoveTensorOptionsRC_2 -> origin/RemoveTensorOptionsRC_2 2022-03-17T19:07:02.7233638Z * [new branch] SsnL-patch-1 -> origin/SsnL-patch-1 2022-03-17T19:07:02.7234917Z * [new branch] SsnL-patch-12 -> origin/SsnL-patch-12 2022-03-17T19:07:02.7236690Z * [new branch] SsnL-patch-4 -> origin/SsnL-patch-4 2022-03-17T19:07:02.7238248Z * [new branch] SsnL-patch-8 -> origin/SsnL-patch-8 2022-03-17T19:07:02.7239580Z * [new branch] T103677508-maxren -> origin/T103677508-maxren 2022-03-17T19:07:02.7241379Z * [new branch] T105015700 -> origin/T105015700 2022-03-17T19:07:02.7242563Z * [new branch] T105015700-2 -> origin/T105015700-2 2022-03-17T19:07:02.7244488Z * [new branch] T105015700-3 -> origin/T105015700-3 2022-03-17T19:07:02.7245677Z * [new branch] T98160523 -> origin/T98160523 2022-03-17T19:07:02.7247189Z * [new branch] T98160523_2 -> origin/T98160523_2 2022-03-17T19:07:02.7248714Z * [new branch] TO_WORK_BRANCH -> origin/TO_WORK_BRANCH 2022-03-17T19:07:02.7250563Z * [new branch] TarFileReaderFix -> origin/TarFileReaderFix 2022-03-17T19:07:02.7251815Z * [new branch] TorchDataDocsUpdate -> origin/TorchDataDocsUpdate 2022-03-17T19:07:02.7253329Z * [new branch] Update-libtorch-path -> origin/Update-libtorch-path 2022-03-17T19:07:02.7254669Z * [new branch] Update-libtorch-path-2 -> origin/Update-libtorch-path-2 2022-03-17T19:07:02.7256112Z * [new branch] Update-test-distribute -> origin/Update-test-distribute 2022-03-17T19:07:02.7257501Z * [new branch] Update-test-rpc -> origin/Update-test-rpc 2022-03-17T19:07:02.7259036Z * [new branch] Update-test-vulkan -> origin/Update-test-vulkan 2022-03-17T19:07:02.7260775Z * [new branch] VitalyFedyunin-patch-1 -> origin/VitalyFedyunin-patch-1 2022-03-17T19:07:02.7263663Z * [new branch] __gg_suo/1603135434 -> origin/__gg_suo/1603135434 2022-03-17T19:07:02.7264841Z * [new branch] __gg_suo/1645154242 -> origin/__gg_suo/1645154242 2022-03-17T19:07:02.7266375Z * [new branch] __gg_suo/1647471426 -> origin/__gg_suo/1647471426 2022-03-17T19:07:02.7267731Z * [new branch] _aminmax_deprecation -> origin/_aminmax_deprecation 2022-03-17T19:07:02.7269721Z * [new branch] activation_functions_i_opinfos -> origin/activation_functions_i_opinfos 2022-03-17T19:07:02.7270979Z * [new branch] adapative_autorange_redo -> origin/adapative_autorange_redo 2022-03-17T19:07:02.7272739Z * [new branch] add-docs-rule-GH1 -> origin/add-docs-rule-GH1 2022-03-17T19:07:02.7274022Z * [new branch] add-print-test-stats-step-to-macos -> origin/add-print-test-stats-step-to-macos 2022-03-17T19:07:02.7275887Z * [new branch] add-test-whl -> origin/add-test-whl 2022-03-17T19:07:02.7277114Z * [new branch] add-wheel-test-job -> origin/add-wheel-test-job 2022-03-17T19:07:02.7278973Z * [new branch] add-wheel-test-job-2 -> origin/add-wheel-test-job-2 2022-03-17T19:07:02.7280244Z * [new branch] add_code_filter_for_TS -> origin/add_code_filter_for_TS 2022-03-17T19:07:02.7282102Z * [new branch] add_default_args -> origin/add_default_args 2022-03-17T19:07:02.7283316Z * [new branch] add_llvm -> origin/add_llvm 2022-03-17T19:07:02.7285186Z * [new branch] add_monkeytype_to_tests -> origin/add_monkeytype_to_tests 2022-03-17T19:07:02.7287064Z * [new branch] add_multiline_support_for_typing_variables -> origin/add_multiline_support_for_typing_variables 2022-03-17T19:07:02.7300503Z * [new branch] add_support_for_optional_tensor -> origin/add_support_for_optional_tensor 2022-03-17T19:07:02.7302332Z * [new branch] add_tags_to_Parameter_dev1 -> origin/add_tags_to_Parameter_dev1 2022-03-17T19:07:02.7304152Z * [new branch] add_tests_for_out -> origin/add_tests_for_out 2022-03-17T19:07:02.7305496Z * [new branch] addcmul_opinfo -> origin/addcmul_opinfo 2022-03-17T19:07:02.7307370Z * [new branch] adjoint -> origin/adjoint 2022-03-17T19:07:02.7309339Z * [new branch] alanwaketan/ddp_fallback -> origin/alanwaketan/ddp_fallback 2022-03-17T19:07:02.7310692Z * [new branch] alanwaketan/grep_opt_params -> origin/alanwaketan/grep_opt_params 2022-03-17T19:07:02.7312235Z * [new branch] alanwaketan/lazy_bench -> origin/alanwaketan/lazy_bench 2022-03-17T19:07:02.7313639Z * [new branch] alanwaketan/mark_step -> origin/alanwaketan/mark_step 2022-03-17T19:07:02.7314934Z * [new branch] alanwaketan/tb -> origin/alanwaketan/tb 2022-03-17T19:07:02.7316778Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2022-03-17T19:07:02.7318113Z * [new branch] albanD-patch-2 -> origin/albanD-patch-2 2022-03-17T19:07:02.7320488Z * [new branch] albanD-patch-3 -> origin/albanD-patch-3 2022-03-17T19:07:02.7323194Z * [new branch] albanD/codeowner-binding -> origin/albanD/codeowner-binding 2022-03-17T19:07:02.7324607Z * [new branch] albanD/contrib_update -> origin/albanD/contrib_update 2022-03-17T19:07:02.7326731Z * [new branch] alband/fix_gil_bug_distributed -> origin/alband/fix_gil_bug_distributed 2022-03-17T19:07:02.7328076Z * [new branch] aliasanalysis -> origin/aliasanalysis 2022-03-17T19:07:02.7329823Z * [new branch] aliases_in_python -> origin/aliases_in_python 2022-03-17T19:07:02.7331188Z * [new branch] all_codegen_ops -> origin/all_codegen_ops 2022-03-17T19:07:02.7332756Z * [new branch] allgathercoalesced -> origin/allgathercoalesced 2022-03-17T19:07:02.7334092Z * [new branch] always_wrap -> origin/always_wrap 2022-03-17T19:07:02.7335615Z * [new branch] andrew-temp -> origin/andrew-temp 2022-03-17T19:07:02.7337154Z * [new branch] android_fbgemm_off -> origin/android_fbgemm_off 2022-03-17T19:07:02.7338526Z * [new branch] android_fbgemm_off_pr -> origin/android_fbgemm_off_pr 2022-03-17T19:07:02.7340303Z * [new branch] android_imageutils_chlast -> origin/android_imageutils_chlast 2022-03-17T19:07:02.7341883Z * [new branch] android_memory_format -> origin/android_memory_format 2022-03-17T19:07:02.7343670Z * [new branch] annotate_parameter_as_tensor -> origin/annotate_parameter_as_tensor 2022-03-17T19:07:02.7344947Z * [new branch] aot-compiler-shape-quant -> origin/aot-compiler-shape-quant 2022-03-17T19:07:02.7346538Z * [new branch] api_release -> origin/api_release 2022-03-17T19:07:02.7347860Z * [new branch] applySelect_exp -> origin/applySelect_exp 2022-03-17T19:07:02.7349850Z * [new branch] as_tensor_docs -> origin/as_tensor_docs 2022-03-17T19:07:02.7351122Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2022-03-17T19:07:02.7352953Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2022-03-17T19:07:02.7354135Z * [new branch] augment_inline_callstack -> origin/augment_inline_callstack 2022-03-17T19:07:02.7356412Z * [new branch] axitkhurana/docs-typo-fix -> origin/axitkhurana/docs-typo-fix 2022-03-17T19:07:02.7358150Z * [new branch] backport_api -> origin/backport_api 2022-03-17T19:07:02.7359526Z * [new branch] backward -> origin/backward 2022-03-17T19:07:02.7361653Z * [new branch] bahuang/test -> origin/bahuang/test 2022-03-17T19:07:02.7362899Z * [new branch] bahuang/test2 -> origin/bahuang/test2 2022-03-17T19:07:02.7364932Z * [new branch] base/1.5 -> origin/base/1.5 2022-03-17T19:07:02.7366213Z * [new branch] base/6f8328e -> origin/base/6f8328e 2022-03-17T19:07:02.7367481Z * [new branch] base/9e6877c -> origin/base/9e6877c 2022-03-17T19:07:02.7369605Z * [new branch] base/b7b481b -> origin/base/b7b481b 2022-03-17T19:07:02.7370850Z * [new branch] batch_norm_folding_traced_modules -> origin/batch_norm_folding_traced_modules 2022-03-17T19:07:02.7373018Z * [new branch] bc_for_torchvision_nonlinearity_fanmode -> origin/bc_for_torchvision_nonlinearity_fanmode 2022-03-17T19:07:02.7374010Z * [new branch] bench_fix -> origin/bench_fix 2022-03-17T19:07:02.7375916Z * [new branch] bert_refactor -> origin/bert_refactor 2022-03-17T19:07:02.7377789Z * [new branch] bertmaher/nncops -> origin/bertmaher/nncops 2022-03-17T19:07:02.7379716Z * [new branch] bf16 -> origin/bf16 2022-03-17T19:07:02.7381095Z * [new branch] bfgsfix -> origin/bfgsfix 2022-03-17T19:07:02.7383157Z * [new branch] bfloat-elementwise2 -> origin/bfloat-elementwise2 2022-03-17T19:07:02.7384630Z * [new branch] binbao/batch_norm -> origin/binbao/batch_norm 2022-03-17T19:07:02.7385929Z * [new branch] binbao/fast_tracing -> origin/binbao/fast_tracing 2022-03-17T19:07:02.7387341Z * [new branch] binbao/index_tensor -> origin/binbao/index_tensor 2022-03-17T19:07:02.7388755Z * [new branch] binbao/measure_tracing -> origin/binbao/measure_tracing 2022-03-17T19:07:02.7389938Z * [new branch] binbao/merge -> origin/binbao/merge 2022-03-17T19:07:02.7391512Z * [new branch] binbao/merge2 -> origin/binbao/merge2 2022-03-17T19:07:02.7393574Z * [new branch] binbao/merge3 -> origin/binbao/merge3 2022-03-17T19:07:02.7394879Z * [new branch] bincount_opinfo -> origin/bincount_opinfo 2022-03-17T19:07:02.7396839Z * [new branch] boo2 -> origin/boo2 2022-03-17T19:07:02.7398185Z * [new branch] bowang_test -> origin/bowang_test 2022-03-17T19:07:02.7400207Z * [new branch] bowbao/extend_onnx_rule_files -> origin/bowbao/extend_onnx_rule_files 2022-03-17T19:07:02.7401335Z * [new branch] bowbao/fix_lstm_regression -> origin/bowbao/fix_lstm_regression 2022-03-17T19:07:02.7402588Z * [new branch] bowbao/onnx_bart -> origin/bowbao/onnx_bart 2022-03-17T19:07:02.7403977Z * [new branch] bowbao/onnx_embedding_renorm -> origin/bowbao/onnx_embedding_renorm 2022-03-17T19:07:02.7405182Z * [new branch] bowbao/onnx_infer_scope_func -> origin/bowbao/onnx_infer_scope_func 2022-03-17T19:07:02.7406886Z * [new branch] bowbao/onnx_local_function_attrs -> origin/bowbao/onnx_local_function_attrs 2022-03-17T19:07:02.7408808Z * [new branch] bowbao/onnx_tracing_len -> origin/bowbao/onnx_tracing_len 2022-03-17T19:07:02.7411248Z * [new branch] boyuantan/print-sccache-only-on-builds -> origin/boyuantan/print-sccache-only-on-builds 2022-03-17T19:07:02.7412303Z * [new branch] break-llvm -> origin/break-llvm 2022-03-17T19:07:02.7414121Z * [new branch] brianjo-doctest -> origin/brianjo-doctest 2022-03-17T19:07:02.7415902Z * [new branch] brianjo-expand-fix -> origin/brianjo-expand-fix 2022-03-17T19:07:02.7417209Z * [new branch] brianjo-jitv2fix -> origin/brianjo-jitv2fix 2022-03-17T19:07:02.7418707Z * [new branch] brianjo-torchrec-domainlist -> origin/brianjo-torchrec-domainlist 2022-03-17T19:07:02.7420668Z * [new branch] brianjo_versions -> origin/brianjo_versions 2022-03-17T19:07:02.7422032Z * [new branch] bucketize_opinfo -> origin/bucketize_opinfo 2022-03-17T19:07:02.7423527Z * [new branch] build-test-whl -> origin/build-test-whl 2022-03-17T19:07:02.7424863Z * [new branch] build-whl-for-testing -> origin/build-whl-for-testing 2022-03-17T19:07:02.7426780Z * [new branch] build-whl-package-2 -> origin/build-whl-package-2 2022-03-17T19:07:02.7427878Z * [new branch] build-whl-package-3 -> origin/build-whl-package-3 2022-03-17T19:07:02.7429410Z * [new branch] build-whl-package-4 -> origin/build-whl-package-4 2022-03-17T19:07:02.7431418Z * [new branch] build/1.3.0 -> origin/build/1.3.0 2022-03-17T19:07:02.7432515Z * [new branch] build/1.3.1 -> origin/build/1.3.1 2022-03-17T19:07:02.7434149Z * [new branch] cached_functions -> origin/cached_functions 2022-03-17T19:07:02.7435474Z * [new branch] cacherator -> origin/cacherator 2022-03-17T19:07:02.7437132Z * [new branch] caffe_2_master_references_to_main -> origin/caffe_2_master_references_to_main 2022-03-17T19:07:02.7438552Z * [new branch] cbalioglu-patch-1 -> origin/cbalioglu-patch-1 2022-03-17T19:07:02.7439914Z * [new branch] change_ns_cuda_mar_10 -> origin/change_ns_cuda_mar_10 2022-03-17T19:07:02.7441779Z * [new branch] check-cupti-so -> origin/check-cupti-so 2022-03-17T19:07:02.7442841Z * [new branch] checkout-pytorch -> origin/checkout-pytorch 2022-03-17T19:07:02.7444931Z * [new branch] cherry-pick-58572 -> origin/cherry-pick-58572 2022-03-17T19:07:02.7446061Z * [new branch] cherry-pick-58573 -> origin/cherry-pick-58573 2022-03-17T19:07:02.7447987Z * [new branch] cherry-pick-fix-refcounting -> origin/cherry-pick-fix-refcounting 2022-03-17T19:07:02.7449195Z * [new branch] cherrypick_xla_codegen -> origin/cherrypick_xla_codegen 2022-03-17T19:07:02.7451731Z * [new branch] chesterliu/dev/fix-warnings-for-msvc-1 -> origin/chesterliu/dev/fix-warnings-for-msvc-1 2022-03-17T19:07:02.7452807Z * [new branch] chesterliu/dev/skip-nccl-1 -> origin/chesterliu/dev/skip-nccl-1 2022-03-17T19:07:02.7454649Z * [new branch] chown -> origin/chown 2022-03-17T19:07:02.7455817Z * [new branch] ci -> origin/ci 2022-03-17T19:07:02.7458052Z * [new branch] ci-all/aar_headers_test_app_2 -> origin/ci-all/aar_headers_test_app_2 2022-03-17T19:07:02.7459058Z * [new branch] ci-all/add-back-11.1-ci -> origin/ci-all/add-back-11.1-ci 2022-03-17T19:07:02.7461025Z * [new branch] ci-all/add-initial-linux-ci-gha -> origin/ci-all/add-initial-linux-ci-gha 2022-03-17T19:07:02.7462575Z * [new branch] ci-all/agolynski/mpicuda -> origin/ci-all/agolynski/mpicuda 2022-03-17T19:07:02.7464515Z * [new branch] ci-all/albanD/fwad_opinfo -> origin/ci-all/albanD/fwad_opinfo 2022-03-17T19:07:02.7466560Z * [new branch] ci-all/android_aar_headers_ci -> origin/ci-all/android_aar_headers_ci 2022-03-17T19:07:02.7467836Z * [new branch] ci-all/awgu -> origin/ci-all/awgu 2022-03-17T19:07:02.7469200Z * [new branch] ci-all/b_06e66e29 -> origin/ci-all/b_06e66e29 2022-03-17T19:07:02.7470564Z * [new branch] ci-all/baddbmm_port -> origin/ci-all/baddbmm_port 2022-03-17T19:07:02.7472193Z * [new branch] ci-all/batchnorm3d-channels-last-cudnn-spatial-persistent -> origin/ci-all/batchnorm3d-channels-last-cudnn-spatial-persistent 2022-03-17T19:07:02.7473319Z * [new branch] ci-all/better-numerics-test -> origin/ci-all/better-numerics-test 2022-03-17T19:07:02.7475331Z * [new branch] ci-all/bfloat16-cuda-tests -> origin/ci-all/bfloat16-cuda-tests 2022-03-17T19:07:02.7495612Z * [new branch] ci-all/binary_debug_test -> origin/ci-all/binary_debug_test 2022-03-17T19:07:02.7497814Z * [new branch] ci-all/ci-all/solve-internal-rewrite -> origin/ci-all/ci-all/solve-internal-rewrite 2022-03-17T19:07:02.7499065Z * [new branch] ci-all/ci-vulkan-test -> origin/ci-all/ci-vulkan-test 2022-03-17T19:07:02.7500763Z * [new branch] ci-all/ci_vulkan_test -> origin/ci-all/ci_vulkan_test 2022-03-17T19:07:02.7502935Z * [new branch] ci-all/conv-double-backward-40999 -> origin/ci-all/conv-double-backward-40999 2022-03-17T19:07:02.7503940Z * [new branch] ci-all/conv3d-channels-last -> origin/ci-all/conv3d-channels-last 2022-03-17T19:07:02.7505401Z * [new branch] ci-all/cublas-triangular-solve -> origin/ci-all/cublas-triangular-solve 2022-03-17T19:07:02.7506793Z * [new branch] ci-all/cuda_111_debug_fix -> origin/ci-all/cuda_111_debug_fix 2022-03-17T19:07:02.7508728Z * [new branch] ci-all/cupti_so -> origin/ci-all/cupti_so 2022-03-17T19:07:02.7509787Z * [new branch] ci-all/cusolver-cholesky-1 -> origin/ci-all/cusolver-cholesky-1 2022-03-17T19:07:02.7511581Z * [new branch] ci-all/cusolver-cholesky-batched_cuda11.3 -> origin/ci-all/cusolver-cholesky-batched_cuda11.3 2022-03-17T19:07:02.7512673Z * [new branch] ci-all/cusolver-cholesky-inverse -> origin/ci-all/cusolver-cholesky-inverse 2022-03-17T19:07:02.7514382Z * [new branch] ci-all/cusolver-cholesky-solve -> origin/ci-all/cusolver-cholesky-solve 2022-03-17T19:07:02.7515561Z * [new branch] ci-all/cusolver-svd -> origin/ci-all/cusolver-svd 2022-03-17T19:07:02.7517155Z * [new branch] ci-all/debug-lmem -> origin/ci-all/debug-lmem 2022-03-17T19:07:02.7518383Z * [new branch] ci-all/debuggable-scheduled-ci -> origin/ci-all/debuggable-scheduled-ci 2022-03-17T19:07:02.7520401Z * [new branch] ci-all/default-dtype -> origin/ci-all/default-dtype 2022-03-17T19:07:02.7521376Z * [new branch] ci-all/dot_vdot_bmm_mv -> origin/ci-all/dot_vdot_bmm_mv 2022-03-17T19:07:02.7522741Z * [new branch] ci-all/enable_pe -> origin/ci-all/enable_pe 2022-03-17T19:07:02.7524119Z * [new branch] ci-all/fbjni_022 -> origin/ci-all/fbjni_022 2022-03-17T19:07:02.7525875Z * [new branch] ci-all/fbjni_022_2 -> origin/ci-all/fbjni_022_2 2022-03-17T19:07:02.7527073Z * [new branch] ci-all/fbjni_022_prefab_2 -> origin/ci-all/fbjni_022_prefab_2 2022-03-17T19:07:02.7529460Z * [new branch] ci-all/fe_test1 -> origin/ci-all/fe_test1 2022-03-17T19:07:02.7530464Z * [new branch] ci-all/fe_test1_fail -> origin/ci-all/fe_test1_fail 2022-03-17T19:07:02.7531821Z * [new branch] ci-all/fe_test1_fail_2 -> origin/ci-all/fe_test1_fail_2 2022-03-17T19:07:02.7533136Z * [new branch] ci-all/fe_test1_fail_2_my -> origin/ci-all/fe_test1_fail_2_my 2022-03-17T19:07:02.7534772Z * [new branch] ci-all/fe_test1_fail_2_my_m -> origin/ci-all/fe_test1_fail_2_my_m 2022-03-17T19:07:02.7535916Z * [new branch] ci-all/fix-54381-and-53976 -> origin/ci-all/fix-54381-and-53976 2022-03-17T19:07:02.7537383Z * [new branch] ci-all/fix-worksize-lstsq-magma -> origin/ci-all/fix-worksize-lstsq-magma 2022-03-17T19:07:02.7538739Z * [new branch] ci-all/fix_cu112_windows -> origin/ci-all/fix_cu112_windows 2022-03-17T19:07:02.7540434Z * [new branch] ci-all/fix_cuda112_jobs -> origin/ci-all/fix_cuda112_jobs 2022-03-17T19:07:02.7541810Z * [new branch] ci-all/gen-py -> origin/ci-all/gen-py 2022-03-17T19:07:02.7544837Z * [new branch] ci-all/gh/smessmer/307/head -> origin/ci-all/gh/smessmer/307/head 2022-03-17T19:07:02.7545861Z * [new branch] ci-all/h-huang -> origin/ci-all/h-huang 2022-03-17T19:07:02.7547689Z * [new branch] ci-all/hackathon_opinfo_matmul_2 -> origin/ci-all/hackathon_opinfo_matmul_2 2022-03-17T19:07:02.7548771Z * [new branch] ci-all/have-ci-all-still-work -> origin/ci-all/have-ci-all-still-work 2022-03-17T19:07:02.7550380Z * [new branch] ci-all/ic_prof_ref -> origin/ci-all/ic_prof_ref 2022-03-17T19:07:02.7551648Z * [new branch] ci-all/kineto_cpu -> origin/ci-all/kineto_cpu 2022-03-17T19:07:02.7553522Z * [new branch] ci-all/kineto_cpu_0303 -> origin/ci-all/kineto_cpu_0303 2022-03-17T19:07:02.7554498Z * [new branch] ci-all/kineto_exp -> origin/ci-all/kineto_exp 2022-03-17T19:07:02.7557336Z * [new branch] ci-all/krovatkin/profiling_fullforce -> origin/ci-all/krovatkin/profiling_fullforce 2022-03-17T19:07:02.7558798Z * [new branch] ci-all/krshrimali/record-time-slowtests -> origin/ci-all/krshrimali/record-time-slowtests 2022-03-17T19:07:02.7560276Z * [new branch] ci-all/mrshen -> origin/ci-all/mrshen 2022-03-17T19:07:02.7561702Z * [new branch] ci-all/mrshenli -> origin/ci-all/mrshenli 2022-03-17T19:07:02.7563603Z * [new branch] ci-all/mvz-ci-all -> origin/ci-all/mvz-ci-all 2022-03-17T19:07:02.7565222Z * [new branch] ci-all/mvz-te-ext-calls -> origin/ci-all/mvz-te-ext-calls 2022-03-17T19:07:02.7567620Z * [new branch] ci-all/orig/release/1.6 -> origin/ci-all/orig/release/1.6 2022-03-17T19:07:02.7568719Z * [new branch] ci-all/orig/release/1.7.1 -> origin/ci-all/orig/release/1.7.1 2022-03-17T19:07:02.7570992Z * [new branch] ci-all/peterbell10/batch-norm-cuda -> origin/ci-all/peterbell10/batch-norm-cuda 2022-03-17T19:07:02.7572248Z * [new branch] ci-all/post_backward_sync -> origin/ci-all/post_backward_sync 2022-03-17T19:07:02.7574028Z * [new branch] ci-all/pr_49201_2 -> origin/ci-all/pr_49201_2 2022-03-17T19:07:02.7576148Z * [new branch] ci-all/pritam/dist_autograd_sync -> origin/ci-all/pritam/dist_autograd_sync 2022-03-17T19:07:02.7577417Z * [new branch] ci-all/pritam/dist_autograd_sync_lastest -> origin/ci-all/pritam/dist_autograd_sync_lastest 2022-03-17T19:07:02.7578780Z * [new branch] ci-all/record-time-slowtests -> origin/ci-all/record-time-slowtests 2022-03-17T19:07:02.7580559Z * [new branch] ci-all/reenable-skipped-tests-for-other-cuda -> origin/ci-all/reenable-skipped-tests-for-other-cuda 2022-03-17T19:07:02.7582018Z * [new branch] ci-all/reenable-windows-standalone -> origin/ci-all/reenable-windows-standalone 2022-03-17T19:07:02.7583552Z * [new branch] ci-all/reenable-windows-test_nn-tests -> origin/ci-all/reenable-windows-test_nn-tests 2022-03-17T19:07:02.7586452Z * [new branch] ci-all/reland/develop/complex/masked_fill -> origin/ci-all/reland/develop/complex/masked_fill 2022-03-17T19:07:02.7587671Z * [new branch] ci-all/reland_addbmm -> origin/ci-all/reland_addbmm 2022-03-17T19:07:02.7589469Z * [new branch] ci-all/remove-thrust_t-arith -> origin/ci-all/remove-thrust_t-arith 2022-03-17T19:07:02.7590819Z * [new branch] ci-all/remove-thrust_t-eqne -> origin/ci-all/remove-thrust_t-eqne 2022-03-17T19:07:02.7592876Z * [new branch] ci-all/replace-linux-ci-11.1-with-11.2 -> origin/ci-all/replace-linux-ci-11.1-with-11.2 2022-03-17T19:07:02.7594080Z * [new branch] ci-all/replace-windows-11.1-ci-with-11.2 -> origin/ci-all/replace-windows-11.1-ci-with-11.2 2022-03-17T19:07:02.7595477Z * [new branch] ci-all/revert-render-test -> origin/ci-all/revert-render-test 2022-03-17T19:07:02.7598222Z * [new branch] ci-all/rohan/flaky -> origin/ci-all/rohan/flaky 2022-03-17T19:07:02.7599276Z * [new branch] ci-all/rohan/gputest -> origin/ci-all/rohan/gputest 2022-03-17T19:07:02.7600978Z * [new branch] ci-all/rohan/pg_wrapper -> origin/ci-all/rohan/pg_wrapper 2022-03-17T19:07:02.7602299Z * [new branch] ci-all/rohan/pg_wrapper_2 -> origin/ci-all/rohan/pg_wrapper_2 2022-03-17T19:07:02.7603894Z * [new branch] ci-all/rohan/pg_wrapper_ci_2 -> origin/ci-all/rohan/pg_wrapper_ci_2 2022-03-17T19:07:02.7605085Z * [new branch] ci-all/rohan/test_autograd_prof -> origin/ci-all/rohan/test_autograd_prof 2022-03-17T19:07:02.7606488Z * [new branch] ci-all/rohan/test_prof -> origin/ci-all/rohan/test_prof 2022-03-17T19:07:02.7608905Z * [new branch] ci-all/rvarm1/ci-all-barrier -> origin/ci-all/rvarm1/ci-all-barrier 2022-03-17T19:07:02.7610164Z * [new branch] ci-all/rvarm1/ci_all_ddp_mult_bwd -> origin/ci-all/rvarm1/ci_all_ddp_mult_bwd 2022-03-17T19:07:02.7611343Z * [new branch] ci-all/rvarm1/ddp_bwd -> origin/ci-all/rvarm1/ddp_bwd 2022-03-17T19:07:02.7613300Z * [new branch] ci-all/rvarm1/ddp_local_fix -> origin/ci-all/rvarm1/ddp_local_fix 2022-03-17T19:07:02.7614231Z * [new branch] ci-all/rvarm1/ddp_multi -> origin/ci-all/rvarm1/ddp_multi 2022-03-17T19:07:02.7615894Z * [new branch] ci-all/rvarm1/dist_optim -> origin/ci-all/rvarm1/dist_optim 2022-03-17T19:07:02.7617831Z * [new branch] ci-all/rvarm1/fix_unused -> origin/ci-all/rvarm1/fix_unused 2022-03-17T19:07:02.7619488Z * [new branch] ci-all/rvarm1/param_name_ddp -> origin/ci-all/rvarm1/param_name_ddp 2022-03-17T19:07:02.7620558Z * [new branch] ci-all/rvarm1/test_script -> origin/ci-all/rvarm1/test_script 2022-03-17T19:07:02.7622193Z * [new branch] ci-all/rvarm1/torch_check -> origin/ci-all/rvarm1/torch_check 2022-03-17T19:07:02.7623181Z * [new branch] ci-all/rvarm1/torch_check_ci_all -> origin/ci-all/rvarm1/torch_check_ci_all 2022-03-17T19:07:02.7625134Z * [new branch] ci-all/shared-cupti -> origin/ci-all/shared-cupti 2022-03-17T19:07:02.7626334Z * [new branch] ci-all/syncbn-channels-lastt-46906 -> origin/ci-all/syncbn-channels-lastt-46906 2022-03-17T19:07:02.7627832Z * [new branch] ci-all/test -> origin/ci-all/test 2022-03-17T19:07:02.7630123Z * [new branch] ci-all/test-time-do-not-merge -> origin/ci-all/test-time-do-not-merge 2022-03-17T19:07:02.7631338Z * [new branch] ci-all/test_all -> origin/ci-all/test_all 2022-03-17T19:07:02.7633356Z * [new branch] ci-all/test_binary_magma_2.5.4 -> origin/ci-all/test_binary_magma_2.5.4 2022-03-17T19:07:02.7634610Z * [new branch] ci-all/test_code_analyzer_regex -> origin/ci-all/test_code_analyzer_regex 2022-03-17T19:07:02.7635992Z * [new branch] ci-all/test_distutils -> origin/ci-all/test_distutils 2022-03-17T19:07:02.7637910Z * [new branch] ci-all/test_early_terminate_cuda_impact_on_test_times -> origin/ci-all/test_early_terminate_cuda_impact_on_test_times 2022-03-17T19:07:02.7638995Z * [new branch] ci-all/test_redispatching -> origin/ci-all/test_redispatching 2022-03-17T19:07:02.7640624Z * [new branch] ci-all/test_terminate_handler -> origin/ci-all/test_terminate_handler 2022-03-17T19:07:02.7641963Z * [new branch] ci-all/torchdeploy-reenable -> origin/ci-all/torchdeploy-reenable 2022-03-17T19:07:02.7643550Z * [new branch] ci-all/upd_k_0609 -> origin/ci-all/upd_k_0609 2022-03-17T19:07:02.7644910Z * [new branch] ci-all/upd_kineto_0512 -> origin/ci-all/upd_kineto_0512 2022-03-17T19:07:02.7646852Z * [new branch] ci-all/use_kineto_2 -> origin/ci-all/use_kineto_2 2022-03-17T19:07:02.7648060Z * [new branch] ci-all/use_kineto_3 -> origin/ci-all/use_kineto_3 2022-03-17T19:07:02.7649898Z * [new branch] ci-all/use_kineto_4 -> origin/ci-all/use_kineto_4 2022-03-17T19:07:02.7651135Z * [new branch] ci-all/use_kineto_5 -> origin/ci-all/use_kineto_5 2022-03-17T19:07:02.7652769Z * [new branch] ci-all/use_kineto_6 -> origin/ci-all/use_kineto_6 2022-03-17T19:07:02.7654019Z * [new branch] ci-all/v0dro-cusolver-lu -> origin/ci-all/v0dro-cusolver-lu 2022-03-17T19:07:02.7655914Z * [new branch] ci-all/v0dro-cusolver-lu-solve -> origin/ci-all/v0dro-cusolver-lu-solve 2022-03-17T19:07:02.7658077Z * [new branch] ci-all/vkuzo/fake_quant_test_20210120 -> origin/ci-all/vkuzo/fake_quant_test_20210120 2022-03-17T19:07:02.7659376Z * [new branch] ci-all/vulkan-android-by-default-ci -> origin/ci-all/vulkan-android-by-default-ci 2022-03-17T19:07:02.7660986Z * [new branch] ci-all/vulkan-android-ci-default -> origin/ci-all/vulkan-android-ci-default 2022-03-17T19:07:02.7663322Z * [new branch] ci-all/walterddr/early_terminate_cuda_jit_take_2 -> origin/ci-all/walterddr/early_terminate_cuda_jit_take_2 2022-03-17T19:07:02.7664320Z * [new branch] ci-all/werror_test -> origin/ci-all/werror_test 2022-03-17T19:07:02.7666222Z * [new branch] ci-all/win_cupti -> origin/ci-all/win_cupti 2022-03-17T19:07:02.7667410Z * [new branch] ci-all/win_kineto -> origin/ci-all/win_kineto 2022-03-17T19:07:02.7669988Z * [new branch] ci-all/yanlizhao/enableStaticGraph -> origin/ci-all/yanlizhao/enableStaticGraph 2022-03-17T19:07:02.7671092Z * [new branch] ci-all/yanlizhao/moveRebuildBucket -> origin/ci-all/yanlizhao/moveRebuildBucket 2022-03-17T19:07:02.7672542Z * [new branch] ci-all/yanlizhao/reduceMemory -> origin/ci-all/yanlizhao/reduceMemory 2022-03-17T19:07:02.7674057Z * [new branch] ci-all/yanlizhao/refactorInitializeBucketView -> origin/ci-all/yanlizhao/refactorInitializeBucketView 2022-03-17T19:07:02.7676291Z * [new branch] ci-gradcheck/forward_ad_gradcheck_ci_all -> origin/ci-gradcheck/forward_ad_gradcheck_ci_all 2022-03-17T19:07:02.7677285Z * [new branch] cifix -> origin/cifix 2022-03-17T19:07:02.7678936Z * [new branch] ciflow-push-trigger -> origin/ciflow-push-trigger 2022-03-17T19:07:02.7680473Z * [new branch] circle_ci_master_to_main -> origin/circle_ci_master_to_main 2022-03-17T19:07:02.7681975Z * [new branch] clean_config -> origin/clean_config 2022-03-17T19:07:02.7683513Z * [new branch] clean_up_monkeytype -> origin/clean_up_monkeytype 2022-03-17T19:07:02.7684760Z * [new branch] cleanup-scalar -> origin/cleanup-scalar 2022-03-17T19:07:02.7686607Z * [new branch] cleanup_submodule -> origin/cleanup_submodule 2022-03-17T19:07:02.7688544Z * [new branch] clee2000/disable-jobs-when-owner-not-pytorch -> origin/clee2000/disable-jobs-when-owner-not-pytorch 2022-03-17T19:07:02.7689427Z * [new branch] cluster_test -> origin/cluster_test 2022-03-17T19:07:02.7691713Z * [new branch] codegen_external -> origin/codegen_external 2022-03-17T19:07:02.7692823Z * [new branch] combine-test-times -> origin/combine-test-times 2022-03-17T19:07:02.7694391Z * [new branch] complex_backward -> origin/complex_backward 2022-03-17T19:07:02.7695715Z * [new branch] complex_support -> origin/complex_support 2022-03-17T19:07:02.7697559Z * [new branch] condition_cuda -> origin/condition_cuda 2022-03-17T19:07:02.7698721Z * [new branch] consolidate_manywheel_build -> origin/consolidate_manywheel_build 2022-03-17T19:07:02.7700107Z * [new branch] constant -> origin/constant 2022-03-17T19:07:02.7702112Z * [new branch] conv_empty_batch -> origin/conv_empty_batch 2022-03-17T19:07:02.7703310Z * [new branch] cpp_advanced_indexing -> origin/cpp_advanced_indexing 2022-03-17T19:07:02.7705092Z * [new branch] cpp_embedding_enum -> origin/cpp_embedding_enum 2022-03-17T19:07:02.7706480Z * [new branch] cpp_extension_subprocess_encoding_fixes -> origin/cpp_extension_subprocess_encoding_fixes 2022-03-17T19:07:02.7707815Z * [new branch] cpp_fix_functional_options_naming -> origin/cpp_fix_functional_options_naming 2022-03-17T19:07:02.7709451Z * [new branch] cpp_functional_non_const_ref -> origin/cpp_functional_non_const_ref 2022-03-17T19:07:02.7710735Z * [new branch] cpp_functional_options -> origin/cpp_functional_options 2022-03-17T19:07:02.7712470Z * [new branch] cpp_functional_options_backup -> origin/cpp_functional_options_backup 2022-03-17T19:07:02.7713704Z * [new branch] cpp_functional_options_backup_patch1 -> origin/cpp_functional_options_backup_patch1 2022-03-17T19:07:02.7714933Z * [new branch] cpp_functional_options_backup_patch2 -> origin/cpp_functional_options_backup_patch2 2022-03-17T19:07:02.7716345Z * [new branch] cpp_functional_options_v2 -> origin/cpp_functional_options_v2 2022-03-17T19:07:02.7718190Z * [new branch] cpp_hooks -> origin/cpp_hooks 2022-03-17T19:07:02.7719232Z * [new branch] cpp_module_options_arg_optional -> origin/cpp_module_options_arg_optional 2022-03-17T19:07:02.7720825Z * [new branch] cpp_nn_layers -> origin/cpp_nn_layers 2022-03-17T19:07:02.7722219Z * [new branch] cpp_parity_test_torch_nn_modules -> origin/cpp_parity_test_torch_nn_modules 2022-03-17T19:07:02.7724299Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing -> origin/cpp_tensor_default_dtype_and_advanced_indexing 2022-03-17T19:07:02.7725986Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing_backup -> origin/cpp_tensor_default_dtype_and_advanced_indexing_backup 2022-03-17T19:07:02.7731090Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing_new -> origin/cpp_tensor_default_dtype_and_advanced_indexing_new 2022-03-17T19:07:02.7731851Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing_new_exp1_backup -> origin/cpp_tensor_default_dtype_and_advanced_indexing_new_exp1_backup 2022-03-17T19:07:02.7732550Z * [new branch] cpp_tensor_default_dtype_and_advanced_indexing_tmp_exp -> origin/cpp_tensor_default_dtype_and_advanced_indexing_tmp_exp 2022-03-17T19:07:02.7733226Z * [new branch] cpp_tensor_default_dtype_for_integer -> origin/cpp_tensor_default_dtype_for_integer 2022-03-17T19:07:02.7733758Z * [new branch] cpp_tensor_indexing -> origin/cpp_tensor_indexing 2022-03-17T19:07:02.7734524Z * [new branch] cpu_fastrnn -> origin/cpu_fastrnn 2022-03-17T19:07:02.7736335Z * [new branch] create-ucc-pg -> origin/create-ucc-pg 2022-03-17T19:07:02.7737373Z * [new branch] cuda-long-matmul -> origin/cuda-long-matmul 2022-03-17T19:07:02.7739342Z * [new branch] cuda11_nightlies -> origin/cuda11_nightlies 2022-03-17T19:07:02.7740993Z * [new branch] cuda_stream_jit -> origin/cuda_stream_jit 2022-03-17T19:07:02.7742276Z * [new branch] cudahookbuilderror -> origin/cudahookbuilderror 2022-03-17T19:07:02.7744303Z * [new branch] cudnn-conv-refactor -> origin/cudnn-conv-refactor 2022-03-17T19:07:02.7745505Z * [new branch] cudnn-workspace-limit -> origin/cudnn-workspace-limit 2022-03-17T19:07:02.7746961Z * [new branch] cummax -> origin/cummax 2022-03-17T19:07:02.7748709Z * [new branch] cummax_cummin -> origin/cummax_cummin 2022-03-17T19:07:02.7749990Z * [new branch] cumulative_trapezoid_review -> origin/cumulative_trapezoid_review 2022-03-17T19:07:02.7752406Z * [new branch] dagitses/#62566/optional-delegate-cmp -> origin/dagitses/#62566/optional-delegate-cmp 2022-03-17T19:07:02.7754283Z * [new branch] davidberard98/ltc-docs -> origin/davidberard98/ltc-docs 2022-03-17T19:07:02.7755405Z * [new branch] davidberard98/ubsan-simpleireval -> origin/davidberard98/ubsan-simpleireval 2022-03-17T19:07:02.7756978Z * [new branch] dbr-conv -> origin/dbr-conv 2022-03-17T19:07:02.7758619Z * [new branch] ddp_with_rpc_unit_test -> origin/ddp_with_rpc_unit_test 2022-03-17T19:07:02.7759784Z * [new branch] debug-cpu -> origin/debug-cpu 2022-03-17T19:07:02.7761740Z * [new branch] debugLvlPull -> origin/debugLvlPull 2022-03-17T19:07:02.7762799Z * [new branch] debugging -> origin/debugging 2022-03-17T19:07:02.7764350Z * [new branch] dec14_cuda_stream -> origin/dec14_cuda_stream 2022-03-17T19:07:02.7765667Z * [new branch] default_argument -> origin/default_argument 2022-03-17T19:07:02.7767219Z * [new branch] default_hooks -> origin/default_hooks 2022-03-17T19:07:02.7769037Z * [new branch] devcon -> origin/devcon 2022-03-17T19:07:02.7770222Z * [new branch] devcontainer -> origin/devcontainer 2022-03-17T19:07:02.7771784Z * [new branch] device_type_testing -> origin/device_type_testing 2022-03-17T19:07:02.7773756Z * [new branch] deyu/remove_using_external_data_format -> origin/deyu/remove_using_external_data_format 2022-03-17T19:07:02.7774921Z * [new branch] dfs-needed -> origin/dfs-needed 2022-03-17T19:07:02.7776815Z * [new branch] diag-embed -> origin/diag-embed 2022-03-17T19:07:02.7778003Z * [new branch] diff1 -> origin/diff1 2022-03-17T19:07:02.7779745Z * [new branch] dirty_nnc_randomized -> origin/dirty_nnc_randomized 2022-03-17T19:07:02.7780913Z * [new branch] disable-test -> origin/disable-test 2022-03-17T19:07:02.7783187Z * [new branch] disable-test-history -> origin/disable-test-history 2022-03-17T19:07:02.7784306Z * [new branch] disable_peer_verify_nodesource -> origin/disable_peer_verify_nodesource 2022-03-17T19:07:02.7785969Z * [new branch] dist-documentation-patch-1.8 -> origin/dist-documentation-patch-1.8 2022-03-17T19:07:02.7787376Z * [new branch] dist-expand-refactor -> origin/dist-expand-refactor 2022-03-17T19:07:02.7788939Z * [new branch] dist-grad -> origin/dist-grad 2022-03-17T19:07:02.7790461Z * [new branch] dist_codeowners -> origin/dist_codeowners 2022-03-17T19:07:02.7792780Z * [new branch] dl_compiler_101 -> origin/dl_compiler_101 2022-03-17T19:07:02.7794081Z * [new branch] doc -> origin/doc 2022-03-17T19:07:02.7795875Z * [new branch] doc_fix -> origin/doc_fix 2022-03-17T19:07:02.7797053Z * [new branch] doc_monitored_barrier -> origin/doc_monitored_barrier 2022-03-17T19:07:02.7798685Z * [new branch] docker-job-chown-workspace -> origin/docker-job-chown-workspace 2022-03-17T19:07:02.7800059Z * [new branch] docs-fix-forward_ad-make_dual -> origin/docs-fix-forward_ad-make_dual 2022-03-17T19:07:02.7801442Z * [new branch] dont_decompose -> origin/dont_decompose 2022-03-17T19:07:02.7803959Z * [new branch] driazati/2test2/1 -> origin/driazati/2test2/1 2022-03-17T19:07:02.7805048Z * [new branch] driazati/2test2/2 -> origin/driazati/2test2/2 2022-03-17T19:07:02.7807211Z * [new branch] driazati/awsretry -> origin/driazati/awsretry 2022-03-17T19:07:02.7808211Z * [new branch] driazati/bp -> origin/driazati/bp 2022-03-17T19:07:02.7809567Z * [new branch] driazati/breakpad -> origin/driazati/breakpad 2022-03-17T19:07:02.7810890Z * [new branch] driazati/breakpad2 -> origin/driazati/breakpad2 2022-03-17T19:07:02.7812204Z * [new branch] driazati/breakpad_cmake -> origin/driazati/breakpad_cmake 2022-03-17T19:07:02.7813549Z * [new branch] driazati/breakpad_cmake2 -> origin/driazati/breakpad_cmake2 2022-03-17T19:07:02.7814813Z * [new branch] driazati/circle -> origin/driazati/circle 2022-03-17T19:07:02.7816093Z * [new branch] driazati/circle2 -> origin/driazati/circle2 2022-03-17T19:07:02.7817594Z * [new branch] driazati/doc_previews_fix -> origin/driazati/doc_previews_fix 2022-03-17T19:07:02.7819583Z * [new branch] driazati/docs/12 -> origin/driazati/docs/12 2022-03-17T19:07:02.7820836Z * [new branch] driazati/docsdet -> origin/driazati/docsdet 2022-03-17T19:07:02.7822549Z * [new branch] driazati/end -> origin/driazati/end 2022-03-17T19:07:02.7823822Z * [new branch] driazati/fastcov -> origin/driazati/fastcov 2022-03-17T19:07:02.7826106Z * [new branch] driazati/fixwarn2 -> origin/driazati/fixwarn2 2022-03-17T19:07:02.7826967Z * [new branch] driazati/focal -> origin/driazati/focal 2022-03-17T19:07:02.7828759Z * [new branch] driazati/github_pin -> origin/driazati/github_pin 2022-03-17T19:07:02.7829813Z * [new branch] driazati/it -> origin/driazati/it 2022-03-17T19:07:02.7831167Z * [new branch] driazati/lazy2 -> origin/driazati/lazy2 2022-03-17T19:07:02.7832561Z * [new branch] driazati/noart -> origin/driazati/noart 2022-03-17T19:07:02.7834523Z * [new branch] driazati/rdsre -> origin/driazati/rdsre 2022-03-17T19:07:02.7860568Z * [new branch] driazati/regenfix -> origin/driazati/regenfix 2022-03-17T19:07:02.7862558Z * [new branch] driazati/reme -> origin/driazati/reme 2022-03-17T19:07:02.7864018Z * [new branch] driazati/remove_bd -> origin/driazati/remove_bd 2022-03-17T19:07:02.7865713Z * [new branch] driazati/remove_breakpad -> origin/driazati/remove_breakpad 2022-03-17T19:07:02.7866885Z * [new branch] driazati/remove_render -> origin/driazati/remove_render 2022-03-17T19:07:02.7868608Z * [new branch] driazati/scc -> origin/driazati/scc 2022-03-17T19:07:02.7869967Z * [new branch] driazati/serialize_api -> origin/driazati/serialize_api 2022-03-17T19:07:02.7871799Z * [new branch] driazati/tdp -> origin/driazati/tdp 2022-03-17T19:07:02.7872971Z * [new branch] driazati/test_rds_proxy -> origin/driazati/test_rds_proxy 2022-03-17T19:07:02.7874370Z * [new branch] driazati/testbranch -> origin/driazati/testbranch 2022-03-17T19:07:02.7876947Z * [new branch] driazati/tester/2 -> origin/driazati/tester/2 2022-03-17T19:07:02.7878234Z * [new branch] driazati/testfailures -> origin/driazati/testfailures 2022-03-17T19:07:02.7879715Z * [new branch] driazati/update_onnx -> origin/driazati/update_onnx 2022-03-17T19:07:02.7881136Z * [new branch] driazati/windows_s3_reports -> origin/driazati/windows_s3_reports 2022-03-17T19:07:02.7882921Z * [new branch] duplicate_of_56029 -> origin/duplicate_of_56029 2022-03-17T19:07:02.7884204Z * [new branch] dynamic_reduce_range -> origin/dynamic_reduce_range 2022-03-17T19:07:02.7886118Z * [new branch] dynamic_shapes -> origin/dynamic_shapes 2022-03-17T19:07:02.7887321Z * [new branch] dynamic_shapes_cpp -> origin/dynamic_shapes_cpp 2022-03-17T19:07:02.7889173Z * [new branch] dynamic_shapes_python -> origin/dynamic_shapes_python 2022-03-17T19:07:02.7890384Z * [new branch] eager_compilation -> origin/eager_compilation 2022-03-17T19:07:02.7892345Z * [new branch] ed02619 -> origin/ed02619 2022-03-17T19:07:02.7893725Z * [new branch] edward-io-patch-1 -> origin/edward-io-patch-1 2022-03-17T19:07:02.7895159Z * [new branch] elementwise_binary_ii_opinfos -> origin/elementwise_binary_ii_opinfos 2022-03-17T19:07:02.7896567Z * [new branch] elementwise_binary_opinfos -> origin/elementwise_binary_opinfos 2022-03-17T19:07:02.7898188Z * [new branch] embedding-cub-2 -> origin/embedding-cub-2 2022-03-17T19:07:02.7899497Z * [new branch] empty -> origin/empty 2022-03-17T19:07:02.7900953Z * [new branch] enable_custom_codefilter -> origin/enable_custom_codefilter 2022-03-17T19:07:02.7902971Z * [new branch] enable_metal_in_pytorch_macos_nightly -> origin/enable_metal_in_pytorch_macos_nightly 2022-03-17T19:07:02.7904471Z * [new branch] enabling_checking_typing_attributes_during_rcb_lookup -> origin/enabling_checking_typing_attributes_during_rcb_lookup 2022-03-17T19:07:02.7905801Z * [new branch] error_check_env_fix -> origin/error_check_env_fix 2022-03-17T19:07:02.7907562Z * [new branch] error_input_tests_to_opinfo -> origin/error_input_tests_to_opinfo 2022-03-17T19:07:02.7908789Z * [new branch] error_on_composite_registrations -> origin/error_on_composite_registrations 2022-03-17T19:07:02.7910931Z * [new branch] exercise1 -> origin/exercise1 2022-03-17T19:07:02.7912464Z * [new branch] export-D14992856 -> origin/export-D14992856 2022-03-17T19:07:02.7914177Z * [new branch] export-D14992857 -> origin/export-D14992857 2022-03-17T19:07:02.7915428Z * [new branch] export-D15634027 -> origin/export-D15634027 2022-03-17T19:07:02.7917133Z * [new branch] export-D15795024 -> origin/export-D15795024 2022-03-17T19:07:02.7918303Z * [new branch] export-D15814743 -> origin/export-D15814743 2022-03-17T19:07:02.7920008Z * [new branch] expr_handle_pow -> origin/expr_handle_pow 2022-03-17T19:07:02.7921242Z * [new branch] fb-config -> origin/fb-config 2022-03-17T19:07:02.7923266Z * [new branch] fbcode/warm -> origin/fbcode/warm 2022-03-17T19:07:02.7924528Z * [new branch] fbjni_022 -> origin/fbjni_022 2022-03-17T19:07:02.7925757Z * [new branch] fbjni_022_2 -> origin/fbjni_022_2 2022-03-17T19:07:02.7927551Z * [new branch] fbjni_022_prefab -> origin/fbjni_022_prefab 2022-03-17T19:07:02.7928501Z * [new branch] fbjni_022_prefab_2 -> origin/fbjni_022_prefab_2 2022-03-17T19:07:02.7930422Z * [new branch] fbsync -> origin/fbsync 2022-03-17T19:07:02.7931721Z * [new branch] fix-cat-transform-docs -> origin/fix-cat-transform-docs 2022-03-17T19:07:02.7933152Z * [new branch] fix-dirichlet-shape -> origin/fix-dirichlet-shape 2022-03-17T19:07:02.7934583Z * [new branch] fix-mac-test-report -> origin/fix-mac-test-report 2022-03-17T19:07:02.7936472Z * [new branch] fix-mac-uploading -> origin/fix-mac-uploading 2022-03-17T19:07:02.7937754Z * [new branch] fix-optim-annotation -> origin/fix-optim-annotation 2022-03-17T19:07:02.7939255Z * [new branch] fix-timer-overflow -> origin/fix-timer-overflow 2022-03-17T19:07:02.7940738Z * [new branch] fix-tools -> origin/fix-tools 2022-03-17T19:07:02.7942626Z * [new branch] fix_conv -> origin/fix_conv 2022-03-17T19:07:02.7943941Z * [new branch] fix_cpp_impl_parity -> origin/fix_cpp_impl_parity 2022-03-17T19:07:02.7945302Z * [new branch] fix_ddp_doc -> origin/fix_ddp_doc 2022-03-17T19:07:02.7947215Z * [new branch] fix_dispatch -> origin/fix_dispatch 2022-03-17T19:07:02.7948300Z * [new branch] fix_doc_issues -> origin/fix_doc_issues 2022-03-17T19:07:02.7949889Z * [new branch] fix_doc_new -> origin/fix_doc_new 2022-03-17T19:07:02.7951220Z * [new branch] fix_docs_Aprl -> origin/fix_docs_Aprl 2022-03-17T19:07:02.7953018Z * [new branch] fix_error_msg -> origin/fix_error_msg 2022-03-17T19:07:02.7954129Z * [new branch] fix_foreach_test -> origin/fix_foreach_test 2022-03-17T19:07:02.7955596Z * [new branch] fix_graphmodule_singleton -> origin/fix_graphmodule_singleton 2022-03-17T19:07:02.7957321Z * [new branch] fix_javadoc_url -> origin/fix_javadoc_url 2022-03-17T19:07:02.7958574Z * [new branch] fix_jit_overview_fut -> origin/fix_jit_overview_fut 2022-03-17T19:07:02.7960605Z * [new branch] fix_module_forward_optional_arg -> origin/fix_module_forward_optional_arg 2022-03-17T19:07:02.7961939Z * [new branch] fix_module_forward_optional_arg_backup -> origin/fix_module_forward_optional_arg_backup 2022-03-17T19:07:02.7963364Z * [new branch] fix_multidim_tensor_ctor -> origin/fix_multidim_tensor_ctor 2022-03-17T19:07:02.7965046Z * [new branch] fix_nccl -> origin/fix_nccl 2022-03-17T19:07:02.7966231Z * [new branch] fix_nccl_2 -> origin/fix_nccl_2 2022-03-17T19:07:02.7967794Z * [new branch] fix_portforward -> origin/fix_portforward 2022-03-17T19:07:02.7969178Z * [new branch] fix_shape_prop -> origin/fix_shape_prop 2022-03-17T19:07:02.7971676Z * [new branch] fix_tensorpipe_cmake_option -> origin/fix_tensorpipe_cmake_option 2022-03-17T19:07:02.7972774Z * [new branch] fix_torch_module_check -> origin/fix_torch_module_check 2022-03-17T19:07:02.7974442Z * [new branch] fix_torch_tensor_ctor_4th_try -> origin/fix_torch_tensor_ctor_4th_try 2022-03-17T19:07:02.7975725Z * [new branch] fix_tp_cma_short_read -> origin/fix_tp_cma_short_read 2022-03-17T19:07:02.7977345Z * [new branch] fixoptionalargs -> origin/fixoptionalargs 2022-03-17T19:07:02.7979049Z * [new branch] flatbuffer -> origin/flatbuffer 2022-03-17T19:07:02.7980535Z * [new branch] float_requant_q8vadd -> origin/float_requant_q8vadd 2022-03-17T19:07:02.7982438Z * [new branch] float_requant_q8vadd_2021 -> origin/float_requant_q8vadd_2021 2022-03-17T19:07:02.7983673Z * [new branch] float_zero_point_qparam_cpu -> origin/float_zero_point_qparam_cpu 2022-03-17T19:07:02.7985864Z * [new branch] flops_print_fix -> origin/flops_print_fix 2022-03-17T19:07:02.7987338Z * [new branch] forbid-inplace-modifi-packing -> origin/forbid-inplace-modifi-packing 2022-03-17T19:07:02.7989297Z * [new branch] foreach_wip -> origin/foreach_wip 2022-03-17T19:07:02.7990435Z * [new branch] forward-fix-bc -> origin/forward-fix-bc 2022-03-17T19:07:02.7992324Z * [new branch] foward -> origin/foward 2022-03-17T19:07:02.7993571Z * [new branch] freeze_module2 -> origin/freeze_module2 2022-03-17T19:07:02.7997336Z * [new branch] freeze_subgraphs -> origin/freeze_subgraphs 2022-03-17T19:07:02.7997797Z * [new branch] freezing -> origin/freezing 2022-03-17T19:07:02.7998190Z * [new branch] fsdp_meta -> origin/fsdp_meta 2022-03-17T19:07:02.7999409Z * [new branch] fx-lowering-refactor -> origin/fx-lowering-refactor 2022-03-17T19:07:02.8001322Z * [new branch] fx-lowering-refactor-2 -> origin/fx-lowering-refactor-2 2022-03-17T19:07:02.8002623Z * [new branch] fx-lowering-refactor-3 -> origin/fx-lowering-refactor-3 2022-03-17T19:07:02.8003979Z * [new branch] fx_buffers -> origin/fx_buffers 2022-03-17T19:07:02.8005856Z * [new branch] fx_codegen -> origin/fx_codegen 2022-03-17T19:07:02.8006989Z * [new branch] fx_opinfo_port -> origin/fx_opinfo_port 2022-03-17T19:07:02.8009202Z * [new branch] garymm/dummy -> origin/garymm/dummy 2022-03-17T19:07:02.8010204Z * [new branch] gatherTopK -> origin/gatherTopK 2022-03-17T19:07:02.8012545Z * [new branch] gelu -> origin/gelu 2022-03-17T19:07:02.8013919Z * [new branch] get-workflow-name -> origin/get-workflow-name 2022-03-17T19:07:02.8016531Z * [new branch] gh/1ntEgr8/1/base -> origin/gh/1ntEgr8/1/base 2022-03-17T19:07:02.8017529Z * [new branch] gh/1ntEgr8/1/head -> origin/gh/1ntEgr8/1/head 2022-03-17T19:07:02.8018977Z * [new branch] gh/1ntEgr8/1/orig -> origin/gh/1ntEgr8/1/orig 2022-03-17T19:07:02.8021056Z * [new branch] gh/1ntEgr8/11/base -> origin/gh/1ntEgr8/11/base 2022-03-17T19:07:02.8022260Z * [new branch] gh/1ntEgr8/11/head -> origin/gh/1ntEgr8/11/head 2022-03-17T19:07:02.8023626Z * [new branch] gh/1ntEgr8/11/orig -> origin/gh/1ntEgr8/11/orig 2022-03-17T19:07:02.8025793Z * [new branch] gh/1ntEgr8/17/base -> origin/gh/1ntEgr8/17/base 2022-03-17T19:07:02.8026773Z * [new branch] gh/1ntEgr8/17/head -> origin/gh/1ntEgr8/17/head 2022-03-17T19:07:02.8028137Z * [new branch] gh/1ntEgr8/17/orig -> origin/gh/1ntEgr8/17/orig 2022-03-17T19:07:02.8030158Z * [new branch] gh/1ntEgr8/18/base -> origin/gh/1ntEgr8/18/base 2022-03-17T19:07:02.8031491Z * [new branch] gh/1ntEgr8/18/head -> origin/gh/1ntEgr8/18/head 2022-03-17T19:07:02.8032766Z * [new branch] gh/1ntEgr8/18/orig -> origin/gh/1ntEgr8/18/orig 2022-03-17T19:07:02.8034615Z * [new branch] gh/1ntEgr8/2/base -> origin/gh/1ntEgr8/2/base 2022-03-17T19:07:02.8035750Z * [new branch] gh/1ntEgr8/2/head -> origin/gh/1ntEgr8/2/head 2022-03-17T19:07:02.8037181Z * [new branch] gh/1ntEgr8/2/orig -> origin/gh/1ntEgr8/2/orig 2022-03-17T19:07:02.8039280Z * [new branch] gh/1ntEgr8/20/base -> origin/gh/1ntEgr8/20/base 2022-03-17T19:07:02.8040333Z * [new branch] gh/1ntEgr8/20/head -> origin/gh/1ntEgr8/20/head 2022-03-17T19:07:02.8041658Z * [new branch] gh/1ntEgr8/20/orig -> origin/gh/1ntEgr8/20/orig 2022-03-17T19:07:02.8043687Z * [new branch] gh/1ntEgr8/22/base -> origin/gh/1ntEgr8/22/base 2022-03-17T19:07:02.8044704Z * [new branch] gh/1ntEgr8/22/head -> origin/gh/1ntEgr8/22/head 2022-03-17T19:07:02.8046038Z * [new branch] gh/1ntEgr8/22/orig -> origin/gh/1ntEgr8/22/orig 2022-03-17T19:07:02.8048083Z * [new branch] gh/1ntEgr8/24/base -> origin/gh/1ntEgr8/24/base 2022-03-17T19:07:02.8049130Z * [new branch] gh/1ntEgr8/24/head -> origin/gh/1ntEgr8/24/head 2022-03-17T19:07:02.8050965Z * [new branch] gh/1ntEgr8/24/orig -> origin/gh/1ntEgr8/24/orig 2022-03-17T19:07:02.8053066Z * [new branch] gh/1ntEgr8/25/base -> origin/gh/1ntEgr8/25/base 2022-03-17T19:07:02.8054106Z * [new branch] gh/1ntEgr8/25/head -> origin/gh/1ntEgr8/25/head 2022-03-17T19:07:02.8055878Z * [new branch] gh/1ntEgr8/25/orig -> origin/gh/1ntEgr8/25/orig 2022-03-17T19:07:02.8056986Z * [new branch] gh/1ntEgr8/3/base -> origin/gh/1ntEgr8/3/base 2022-03-17T19:07:02.8058322Z * [new branch] gh/1ntEgr8/3/head -> origin/gh/1ntEgr8/3/head 2022-03-17T19:07:02.8059670Z * [new branch] gh/1ntEgr8/3/orig -> origin/gh/1ntEgr8/3/orig 2022-03-17T19:07:02.8062336Z * [new branch] gh/Amyh11325/10/base -> origin/gh/Amyh11325/10/base 2022-03-17T19:07:02.8063672Z * [new branch] gh/Amyh11325/10/head -> origin/gh/Amyh11325/10/head 2022-03-17T19:07:02.8065194Z * [new branch] gh/Amyh11325/10/orig -> origin/gh/Amyh11325/10/orig 2022-03-17T19:07:02.8067386Z * [new branch] gh/Amyh11325/13/base -> origin/gh/Amyh11325/13/base 2022-03-17T19:07:02.8068619Z * [new branch] gh/Amyh11325/13/head -> origin/gh/Amyh11325/13/head 2022-03-17T19:07:02.8070386Z * [new branch] gh/Amyh11325/13/orig -> origin/gh/Amyh11325/13/orig 2022-03-17T19:07:02.8072151Z * [new branch] gh/Amyh11325/18/base -> origin/gh/Amyh11325/18/base 2022-03-17T19:07:02.8073409Z * [new branch] gh/Amyh11325/18/head -> origin/gh/Amyh11325/18/head 2022-03-17T19:07:02.8075025Z * [new branch] gh/Amyh11325/18/orig -> origin/gh/Amyh11325/18/orig 2022-03-17T19:07:02.8077174Z * [new branch] gh/AshkanAliabadi/1/base -> origin/gh/AshkanAliabadi/1/base 2022-03-17T19:07:02.8078495Z * [new branch] gh/AshkanAliabadi/1/head -> origin/gh/AshkanAliabadi/1/head 2022-03-17T19:07:02.8079870Z * [new branch] gh/AshkanAliabadi/1/orig -> origin/gh/AshkanAliabadi/1/orig 2022-03-17T19:07:02.8081855Z * [new branch] gh/AshkanAliabadi/30/base -> origin/gh/AshkanAliabadi/30/base 2022-03-17T19:07:02.8082994Z * [new branch] gh/AshkanAliabadi/30/head -> origin/gh/AshkanAliabadi/30/head 2022-03-17T19:07:02.8084342Z * [new branch] gh/AshkanAliabadi/30/orig -> origin/gh/AshkanAliabadi/30/orig 2022-03-17T19:07:02.8086086Z * [new branch] gh/AshkanAliabadi/31/base -> origin/gh/AshkanAliabadi/31/base 2022-03-17T19:07:02.8087183Z * [new branch] gh/AshkanAliabadi/31/head -> origin/gh/AshkanAliabadi/31/head 2022-03-17T19:07:02.8088950Z * [new branch] gh/AshkanAliabadi/31/orig -> origin/gh/AshkanAliabadi/31/orig 2022-03-17T19:07:02.8090390Z * [new branch] gh/AshkanAliabadi/32/base -> origin/gh/AshkanAliabadi/32/base 2022-03-17T19:07:02.8091703Z * [new branch] gh/AshkanAliabadi/32/head -> origin/gh/AshkanAliabadi/32/head 2022-03-17T19:07:02.8093135Z * [new branch] gh/AshkanAliabadi/32/orig -> origin/gh/AshkanAliabadi/32/orig 2022-03-17T19:07:02.8094850Z * [new branch] gh/AshkanAliabadi/33/base -> origin/gh/AshkanAliabadi/33/base 2022-03-17T19:07:02.8095725Z * [new branch] gh/AshkanAliabadi/33/head -> origin/gh/AshkanAliabadi/33/head 2022-03-17T19:07:02.8097139Z * [new branch] gh/AshkanAliabadi/33/orig -> origin/gh/AshkanAliabadi/33/orig 2022-03-17T19:07:02.8099374Z * [new branch] gh/AshkanAliabadi/38/base -> origin/gh/AshkanAliabadi/38/base 2022-03-17T19:07:02.8100609Z * [new branch] gh/AshkanAliabadi/38/head -> origin/gh/AshkanAliabadi/38/head 2022-03-17T19:07:02.8102020Z * [new branch] gh/AshkanAliabadi/38/orig -> origin/gh/AshkanAliabadi/38/orig 2022-03-17T19:07:02.8104104Z * [new branch] gh/AshkanAliabadi/4/base -> origin/gh/AshkanAliabadi/4/base 2022-03-17T19:07:02.8105812Z * [new branch] gh/AshkanAliabadi/46/base -> origin/gh/AshkanAliabadi/46/base 2022-03-17T19:07:02.8107183Z * [new branch] gh/AshkanAliabadi/46/head -> origin/gh/AshkanAliabadi/46/head 2022-03-17T19:07:02.8108906Z * [new branch] gh/AshkanAliabadi/5/base -> origin/gh/AshkanAliabadi/5/base 2022-03-17T19:07:02.8110876Z * [new branch] gh/AshkanAliabadi/50/base -> origin/gh/AshkanAliabadi/50/base 2022-03-17T19:07:02.8112057Z * [new branch] gh/AshkanAliabadi/50/head -> origin/gh/AshkanAliabadi/50/head 2022-03-17T19:07:02.8114415Z * [new branch] gh/AshkanAliabadi/50/orig -> origin/gh/AshkanAliabadi/50/orig 2022-03-17T19:07:02.8116308Z * [new branch] gh/AshkanAliabadi/54/base -> origin/gh/AshkanAliabadi/54/base 2022-03-17T19:07:02.8117385Z * [new branch] gh/AshkanAliabadi/54/head -> origin/gh/AshkanAliabadi/54/head 2022-03-17T19:07:02.8118757Z * [new branch] gh/AshkanAliabadi/54/orig -> origin/gh/AshkanAliabadi/54/orig 2022-03-17T19:07:02.8120765Z * [new branch] gh/AshkanAliabadi/57/base -> origin/gh/AshkanAliabadi/57/base 2022-03-17T19:07:02.8121379Z * [new branch] gh/AshkanAliabadi/57/head -> origin/gh/AshkanAliabadi/57/head 2022-03-17T19:07:02.8122814Z * [new branch] gh/AshkanAliabadi/57/orig -> origin/gh/AshkanAliabadi/57/orig 2022-03-17T19:07:02.8124941Z * [new branch] gh/AshkanAliabadi/59/base -> origin/gh/AshkanAliabadi/59/base 2022-03-17T19:07:02.8126163Z * [new branch] gh/AshkanAliabadi/59/head -> origin/gh/AshkanAliabadi/59/head 2022-03-17T19:07:02.8127485Z * [new branch] gh/AshkanAliabadi/59/orig -> origin/gh/AshkanAliabadi/59/orig 2022-03-17T19:07:02.8129370Z * [new branch] gh/AshkanAliabadi/6/base -> origin/gh/AshkanAliabadi/6/base 2022-03-17T19:07:02.8130782Z * [new branch] gh/AshkanAliabadi/60/base -> origin/gh/AshkanAliabadi/60/base 2022-03-17T19:07:02.8132011Z * [new branch] gh/AshkanAliabadi/60/head -> origin/gh/AshkanAliabadi/60/head 2022-03-17T19:07:02.8133404Z * [new branch] gh/AshkanAliabadi/60/orig -> origin/gh/AshkanAliabadi/60/orig 2022-03-17T19:07:02.8135514Z * [new branch] gh/AshkanAliabadi/64/base -> origin/gh/AshkanAliabadi/64/base 2022-03-17T19:07:02.8136529Z * [new branch] gh/AshkanAliabadi/64/head -> origin/gh/AshkanAliabadi/64/head 2022-03-17T19:07:02.8137853Z * [new branch] gh/AshkanAliabadi/64/orig -> origin/gh/AshkanAliabadi/64/orig 2022-03-17T19:07:02.8140016Z * [new branch] gh/AshkanAliabadi/65/base -> origin/gh/AshkanAliabadi/65/base 2022-03-17T19:07:02.8141030Z * [new branch] gh/AshkanAliabadi/65/head -> origin/gh/AshkanAliabadi/65/head 2022-03-17T19:07:02.8142978Z * [new branch] gh/AshkanAliabadi/65/orig -> origin/gh/AshkanAliabadi/65/orig 2022-03-17T19:07:02.8145683Z * [new branch] gh/AshkanAliabadi/66/base -> origin/gh/AshkanAliabadi/66/base 2022-03-17T19:07:02.8146777Z * [new branch] gh/AshkanAliabadi/66/head -> origin/gh/AshkanAliabadi/66/head 2022-03-17T19:07:02.8148201Z * [new branch] gh/AshkanAliabadi/66/orig -> origin/gh/AshkanAliabadi/66/orig 2022-03-17T19:07:02.8150305Z * [new branch] gh/AshkanAliabadi/67/base -> origin/gh/AshkanAliabadi/67/base 2022-03-17T19:07:02.8151403Z * [new branch] gh/AshkanAliabadi/67/head -> origin/gh/AshkanAliabadi/67/head 2022-03-17T19:07:02.8152759Z * [new branch] gh/AshkanAliabadi/67/orig -> origin/gh/AshkanAliabadi/67/orig 2022-03-17T19:07:02.8154737Z * [new branch] gh/AshkanAliabadi/68/base -> origin/gh/AshkanAliabadi/68/base 2022-03-17T19:07:02.8155666Z * [new branch] gh/AshkanAliabadi/68/head -> origin/gh/AshkanAliabadi/68/head 2022-03-17T19:07:02.8157014Z * [new branch] gh/AshkanAliabadi/68/orig -> origin/gh/AshkanAliabadi/68/orig 2022-03-17T19:07:02.8159422Z * [new branch] gh/BowenBao/109/base -> origin/gh/BowenBao/109/base 2022-03-17T19:07:02.8160566Z * [new branch] gh/BowenBao/109/head -> origin/gh/BowenBao/109/head 2022-03-17T19:07:02.8161912Z * [new branch] gh/BowenBao/109/orig -> origin/gh/BowenBao/109/orig 2022-03-17T19:07:02.8164216Z * [new branch] gh/BowenBao/110/base -> origin/gh/BowenBao/110/base 2022-03-17T19:07:02.8165287Z * [new branch] gh/BowenBao/110/head -> origin/gh/BowenBao/110/head 2022-03-17T19:07:02.8166964Z * [new branch] gh/BowenBao/111/base -> origin/gh/BowenBao/111/base 2022-03-17T19:07:02.8168126Z * [new branch] gh/BowenBao/111/head -> origin/gh/BowenBao/111/head 2022-03-17T19:07:02.8169854Z * [new branch] gh/BowenBao/112/base -> origin/gh/BowenBao/112/base 2022-03-17T19:07:02.8170954Z * [new branch] gh/BowenBao/112/head -> origin/gh/BowenBao/112/head 2022-03-17T19:07:02.8172762Z * [new branch] gh/BowenBao/113/base -> origin/gh/BowenBao/113/base 2022-03-17T19:07:02.8174142Z * [new branch] gh/BowenBao/113/head -> origin/gh/BowenBao/113/head 2022-03-17T19:07:02.8176447Z * [new branch] gh/BowenBao/114/base -> origin/gh/BowenBao/114/base 2022-03-17T19:07:02.8177618Z * [new branch] gh/BowenBao/114/head -> origin/gh/BowenBao/114/head 2022-03-17T19:07:02.8179349Z * [new branch] gh/BowenBao/115/base -> origin/gh/BowenBao/115/base 2022-03-17T19:07:02.8180689Z * [new branch] gh/BowenBao/115/head -> origin/gh/BowenBao/115/head 2022-03-17T19:07:02.8183196Z * [new branch] gh/BowenBao/116/base -> origin/gh/BowenBao/116/base 2022-03-17T19:07:02.8185124Z * [new branch] gh/BowenBao/116/head -> origin/gh/BowenBao/116/head 2022-03-17T19:07:02.8186188Z * [new branch] gh/BowenBao/117/base -> origin/gh/BowenBao/117/base 2022-03-17T19:07:02.8187541Z * [new branch] gh/BowenBao/117/head -> origin/gh/BowenBao/117/head 2022-03-17T19:07:02.8189213Z * [new branch] gh/BowenBao/118/base -> origin/gh/BowenBao/118/base 2022-03-17T19:07:02.8190447Z * [new branch] gh/BowenBao/118/head -> origin/gh/BowenBao/118/head 2022-03-17T19:07:02.8192845Z * [new branch] gh/BowenBao/119/base -> origin/gh/BowenBao/119/base 2022-03-17T19:07:02.8193908Z * [new branch] gh/BowenBao/119/head -> origin/gh/BowenBao/119/head 2022-03-17T19:07:02.8195570Z * [new branch] gh/BowenBao/120/base -> origin/gh/BowenBao/120/base 2022-03-17T19:07:02.8196716Z * [new branch] gh/BowenBao/120/head -> origin/gh/BowenBao/120/head 2022-03-17T19:07:02.8198388Z * [new branch] gh/BowenBao/121/base -> origin/gh/BowenBao/121/base 2022-03-17T19:07:02.8200233Z * [new branch] gh/BowenBao/121/head -> origin/gh/BowenBao/121/head 2022-03-17T19:07:02.8201334Z * [new branch] gh/BowenBao/122/base -> origin/gh/BowenBao/122/base 2022-03-17T19:07:02.8202669Z * [new branch] gh/BowenBao/122/head -> origin/gh/BowenBao/122/head 2022-03-17T19:07:02.8204319Z * [new branch] gh/BowenBao/123/base -> origin/gh/BowenBao/123/base 2022-03-17T19:07:02.8205473Z * [new branch] gh/BowenBao/123/head -> origin/gh/BowenBao/123/head 2022-03-17T19:07:02.8207084Z * [new branch] gh/BowenBao/124/base -> origin/gh/BowenBao/124/base 2022-03-17T19:07:02.8208272Z * [new branch] gh/BowenBao/124/head -> origin/gh/BowenBao/124/head 2022-03-17T19:07:02.8209939Z * [new branch] gh/BowenBao/125/base -> origin/gh/BowenBao/125/base 2022-03-17T19:07:02.8211276Z * [new branch] gh/BowenBao/125/head -> origin/gh/BowenBao/125/head 2022-03-17T19:07:02.8212915Z * [new branch] gh/BowenBao/126/base -> origin/gh/BowenBao/126/base 2022-03-17T19:07:02.8214068Z * [new branch] gh/BowenBao/126/head -> origin/gh/BowenBao/126/head 2022-03-17T19:07:02.8216042Z * [new branch] gh/BowenBao/127/base -> origin/gh/BowenBao/127/base 2022-03-17T19:07:02.8217073Z * [new branch] gh/BowenBao/127/head -> origin/gh/BowenBao/127/head 2022-03-17T19:07:02.8218940Z * [new branch] gh/BowenBao/128/base -> origin/gh/BowenBao/128/base 2022-03-17T19:07:02.8219970Z * [new branch] gh/BowenBao/128/head -> origin/gh/BowenBao/128/head 2022-03-17T19:07:02.8222027Z * [new branch] gh/BowenBao/129/base -> origin/gh/BowenBao/129/base 2022-03-17T19:07:02.8223043Z * [new branch] gh/BowenBao/129/head -> origin/gh/BowenBao/129/head 2022-03-17T19:07:02.8224764Z * [new branch] gh/BowenBao/130/base -> origin/gh/BowenBao/130/base 2022-03-17T19:07:02.8225956Z * [new branch] gh/BowenBao/130/head -> origin/gh/BowenBao/130/head 2022-03-17T19:07:02.8227702Z * [new branch] gh/BowenBao/131/base -> origin/gh/BowenBao/131/base 2022-03-17T19:07:02.8228851Z * [new branch] gh/BowenBao/131/head -> origin/gh/BowenBao/131/head 2022-03-17T19:07:02.8230956Z * [new branch] gh/BowenBao/148/base -> origin/gh/BowenBao/148/base 2022-03-17T19:07:02.8231988Z * [new branch] gh/BowenBao/148/head -> origin/gh/BowenBao/148/head 2022-03-17T19:07:02.8233934Z * [new branch] gh/BowenBao/148/orig -> origin/gh/BowenBao/148/orig 2022-03-17T19:07:02.8235317Z * [new branch] gh/BowenBao/149/base -> origin/gh/BowenBao/149/base 2022-03-17T19:07:02.8236658Z * [new branch] gh/BowenBao/149/head -> origin/gh/BowenBao/149/head 2022-03-17T19:07:02.8238020Z * [new branch] gh/BowenBao/149/orig -> origin/gh/BowenBao/149/orig 2022-03-17T19:07:02.8240113Z * [new branch] gh/BowenBao/150/base -> origin/gh/BowenBao/150/base 2022-03-17T19:07:02.8241238Z * [new branch] gh/BowenBao/150/head -> origin/gh/BowenBao/150/head 2022-03-17T19:07:02.8242693Z * [new branch] gh/BowenBao/150/orig -> origin/gh/BowenBao/150/orig 2022-03-17T19:07:02.8244812Z * [new branch] gh/BowenBao/151/base -> origin/gh/BowenBao/151/base 2022-03-17T19:07:02.8245817Z * [new branch] gh/BowenBao/151/orig -> origin/gh/BowenBao/151/orig 2022-03-17T19:07:02.8248143Z * [new branch] gh/DeepDriving/12/base -> origin/gh/DeepDriving/12/base 2022-03-17T19:07:02.8249270Z * [new branch] gh/DeepDriving/12/head -> origin/gh/DeepDriving/12/head 2022-03-17T19:07:02.8250673Z * [new branch] gh/DeepDriving/12/orig -> origin/gh/DeepDriving/12/orig 2022-03-17T19:07:02.8253043Z * [new branch] gh/DeepDriving/2/base -> origin/gh/DeepDriving/2/base 2022-03-17T19:07:02.8254069Z * [new branch] gh/DeepDriving/2/head -> origin/gh/DeepDriving/2/head 2022-03-17T19:07:02.8255490Z * [new branch] gh/DeepDriving/2/orig -> origin/gh/DeepDriving/2/orig 2022-03-17T19:07:02.8256997Z * [new branch] gh/DeepDriving/3/base -> origin/gh/DeepDriving/3/base 2022-03-17T19:07:02.8258293Z * [new branch] gh/DeepDriving/3/head -> origin/gh/DeepDriving/3/head 2022-03-17T19:07:02.8259669Z * [new branch] gh/DeepDriving/3/orig -> origin/gh/DeepDriving/3/orig 2022-03-17T19:07:02.8261181Z * [new branch] gh/DeepDriving/4/base -> origin/gh/DeepDriving/4/base 2022-03-17T19:07:02.8263217Z * [new branch] gh/DeepDriving/4/head -> origin/gh/DeepDriving/4/head 2022-03-17T19:07:02.8264142Z * [new branch] gh/DeepDriving/4/orig -> origin/gh/DeepDriving/4/orig 2022-03-17T19:07:02.8265858Z * [new branch] gh/DeepDriving/5/base -> origin/gh/DeepDriving/5/base 2022-03-17T19:07:02.8266999Z * [new branch] gh/DeepDriving/5/head -> origin/gh/DeepDriving/5/head 2022-03-17T19:07:02.8268351Z * [new branch] gh/DeepDriving/5/orig -> origin/gh/DeepDriving/5/orig 2022-03-17T19:07:02.8270641Z * [new branch] gh/DeepDriving/7/base -> origin/gh/DeepDriving/7/base 2022-03-17T19:07:02.8271625Z * [new branch] gh/DeepDriving/7/head -> origin/gh/DeepDriving/7/head 2022-03-17T19:07:02.8273321Z * [new branch] gh/DeepDriving/8/base -> origin/gh/DeepDriving/8/base 2022-03-17T19:07:02.8274494Z * [new branch] gh/DeepDriving/8/head -> origin/gh/DeepDriving/8/head 2022-03-17T19:07:02.8275874Z * [new branch] gh/DeepDriving/8/orig -> origin/gh/DeepDriving/8/orig 2022-03-17T19:07:02.8277975Z * [new branch] gh/DeepDriving/9/base -> origin/gh/DeepDriving/9/base 2022-03-17T19:07:02.8279048Z * [new branch] gh/DeepDriving/9/head -> origin/gh/DeepDriving/9/head 2022-03-17T19:07:02.8280384Z * [new branch] gh/DeepDriving/9/orig -> origin/gh/DeepDriving/9/orig 2022-03-17T19:07:02.8282751Z * [new branch] gh/Feey0/26/base -> origin/gh/Feey0/26/base 2022-03-17T19:07:02.8283737Z * [new branch] gh/Feey0/26/head -> origin/gh/Feey0/26/head 2022-03-17T19:07:02.8285050Z * [new branch] gh/Feey0/26/orig -> origin/gh/Feey0/26/orig 2022-03-17T19:07:02.8287144Z * [new branch] gh/Feey0/28/base -> origin/gh/Feey0/28/base 2022-03-17T19:07:02.8288522Z * [new branch] gh/Feey0/28/head -> origin/gh/Feey0/28/head 2022-03-17T19:07:02.8289815Z * [new branch] gh/Feey0/28/orig -> origin/gh/Feey0/28/orig 2022-03-17T19:07:02.8291910Z * [new branch] gh/Feey0/29/base -> origin/gh/Feey0/29/base 2022-03-17T19:07:02.8292974Z * [new branch] gh/Feey0/29/head -> origin/gh/Feey0/29/head 2022-03-17T19:07:02.8294299Z * [new branch] gh/Feey0/29/orig -> origin/gh/Feey0/29/orig 2022-03-17T19:07:02.8296324Z * [new branch] gh/Feey0/3/base -> origin/gh/Feey0/3/base 2022-03-17T19:07:02.8297457Z * [new branch] gh/Feey0/3/head -> origin/gh/Feey0/3/head 2022-03-17T19:07:02.8298758Z * [new branch] gh/Feey0/3/orig -> origin/gh/Feey0/3/orig 2022-03-17T19:07:02.8301182Z * [new branch] gh/H-Huang/21/base -> origin/gh/H-Huang/21/base 2022-03-17T19:07:02.8302339Z * [new branch] gh/H-Huang/21/head -> origin/gh/H-Huang/21/head 2022-03-17T19:07:02.8304034Z * [new branch] gh/H-Huang/21/orig -> origin/gh/H-Huang/21/orig 2022-03-17T19:07:02.8305425Z * [new branch] gh/H-Huang/25/base -> origin/gh/H-Huang/25/base 2022-03-17T19:07:02.8307450Z * [new branch] gh/H-Huang/25/head -> origin/gh/H-Huang/25/head 2022-03-17T19:07:02.8308456Z * [new branch] gh/H-Huang/25/orig -> origin/gh/H-Huang/25/orig 2022-03-17T19:07:02.8310529Z * [new branch] gh/H-Huang/36/base -> origin/gh/H-Huang/36/base 2022-03-17T19:07:02.8311506Z * [new branch] gh/H-Huang/36/head -> origin/gh/H-Huang/36/head 2022-03-17T19:07:02.8312885Z * [new branch] gh/H-Huang/36/orig -> origin/gh/H-Huang/36/orig 2022-03-17T19:07:02.8314953Z * [new branch] gh/H-Huang/39/base -> origin/gh/H-Huang/39/base 2022-03-17T19:07:02.8316092Z * [new branch] gh/H-Huang/39/head -> origin/gh/H-Huang/39/head 2022-03-17T19:07:02.8317433Z * [new branch] gh/H-Huang/39/orig -> origin/gh/H-Huang/39/orig 2022-03-17T19:07:02.8319121Z * [new branch] gh/H-Huang/40/base -> origin/gh/H-Huang/40/base 2022-03-17T19:07:02.8320348Z * [new branch] gh/H-Huang/40/head -> origin/gh/H-Huang/40/head 2022-03-17T19:07:02.8321815Z * [new branch] gh/H-Huang/40/orig -> origin/gh/H-Huang/40/orig 2022-03-17T19:07:02.8324489Z * [new branch] gh/H-Huang/41/base -> origin/gh/H-Huang/41/base 2022-03-17T19:07:02.8325641Z * [new branch] gh/H-Huang/41/head -> origin/gh/H-Huang/41/head 2022-03-17T19:07:02.8327668Z * [new branch] gh/H-Huang/41/orig -> origin/gh/H-Huang/41/orig 2022-03-17T19:07:02.8329685Z * [new branch] gh/H-Huang/43/base -> origin/gh/H-Huang/43/base 2022-03-17T19:07:02.8330799Z * [new branch] gh/H-Huang/43/head -> origin/gh/H-Huang/43/head 2022-03-17T19:07:02.8332502Z * [new branch] gh/H-Huang/43/orig -> origin/gh/H-Huang/43/orig 2022-03-17T19:07:02.8333934Z * [new branch] gh/H-Huang/44/base -> origin/gh/H-Huang/44/base 2022-03-17T19:07:02.8335157Z * [new branch] gh/H-Huang/44/head -> origin/gh/H-Huang/44/head 2022-03-17T19:07:02.8336351Z * [new branch] gh/H-Huang/44/orig -> origin/gh/H-Huang/44/orig 2022-03-17T19:07:02.8338556Z * [new branch] gh/H-Huang/47/base -> origin/gh/H-Huang/47/base 2022-03-17T19:07:02.8339562Z * [new branch] gh/H-Huang/47/head -> origin/gh/H-Huang/47/head 2022-03-17T19:07:02.8340974Z * [new branch] gh/H-Huang/47/orig -> origin/gh/H-Huang/47/orig 2022-03-17T19:07:02.8343717Z * [new branch] gh/H-Huang/48/base -> origin/gh/H-Huang/48/base 2022-03-17T19:07:02.8345825Z * [new branch] gh/H-Huang/48/head -> origin/gh/H-Huang/48/head 2022-03-17T19:07:02.8346864Z * [new branch] gh/H-Huang/48/orig -> origin/gh/H-Huang/48/orig 2022-03-17T19:07:02.8349044Z * [new branch] gh/H-Huang/49/base -> origin/gh/H-Huang/49/base 2022-03-17T19:07:02.8350223Z * [new branch] gh/H-Huang/49/head -> origin/gh/H-Huang/49/head 2022-03-17T19:07:02.8351593Z * [new branch] gh/H-Huang/49/orig -> origin/gh/H-Huang/49/orig 2022-03-17T19:07:02.8353251Z * [new branch] gh/H-Huang/50/base -> origin/gh/H-Huang/50/base 2022-03-17T19:07:02.8354643Z * [new branch] gh/H-Huang/50/head -> origin/gh/H-Huang/50/head 2022-03-17T19:07:02.8355996Z * [new branch] gh/H-Huang/50/orig -> origin/gh/H-Huang/50/orig 2022-03-17T19:07:02.8357843Z * [new branch] gh/H-Huang/51/base -> origin/gh/H-Huang/51/base 2022-03-17T19:07:02.8358976Z * [new branch] gh/H-Huang/51/head -> origin/gh/H-Huang/51/head 2022-03-17T19:07:02.8360331Z * [new branch] gh/H-Huang/51/orig -> origin/gh/H-Huang/51/orig 2022-03-17T19:07:02.8361922Z * [new branch] gh/H-Huang/52/base -> origin/gh/H-Huang/52/base 2022-03-17T19:07:02.8363174Z * [new branch] gh/H-Huang/52/head -> origin/gh/H-Huang/52/head 2022-03-17T19:07:02.8364543Z * [new branch] gh/H-Huang/52/orig -> origin/gh/H-Huang/52/orig 2022-03-17T19:07:02.8367062Z * [new branch] gh/HDCharles/3/base -> origin/gh/HDCharles/3/base 2022-03-17T19:07:02.8368144Z * [new branch] gh/HDCharles/3/head -> origin/gh/HDCharles/3/head 2022-03-17T19:07:02.8369599Z * [new branch] gh/HDCharles/3/orig -> origin/gh/HDCharles/3/orig 2022-03-17T19:07:02.8370967Z * [new branch] gh/HDCharles/4/base -> origin/gh/HDCharles/4/base 2022-03-17T19:07:02.8372581Z * [new branch] gh/HDCharles/4/head -> origin/gh/HDCharles/4/head 2022-03-17T19:07:02.8373813Z * [new branch] gh/HDCharles/4/orig -> origin/gh/HDCharles/4/orig 2022-03-17T19:07:02.8375962Z * [new branch] gh/HDCharles/40/base -> origin/gh/HDCharles/40/base 2022-03-17T19:07:02.8377247Z * [new branch] gh/HDCharles/40/head -> origin/gh/HDCharles/40/head 2022-03-17T19:07:02.8379169Z * [new branch] gh/HDCharles/40/orig -> origin/gh/HDCharles/40/orig 2022-03-17T19:07:02.8381282Z * [new branch] gh/HDCharles/41/base -> origin/gh/HDCharles/41/base 2022-03-17T19:07:02.8383405Z * [new branch] gh/HDCharles/41/head -> origin/gh/HDCharles/41/head 2022-03-17T19:07:02.8384313Z * [new branch] gh/HDCharles/41/orig -> origin/gh/HDCharles/41/orig 2022-03-17T19:07:02.8386322Z * [new branch] gh/HDCharles/42/base -> origin/gh/HDCharles/42/base 2022-03-17T19:07:02.8387413Z * [new branch] gh/HDCharles/42/head -> origin/gh/HDCharles/42/head 2022-03-17T19:07:02.8388840Z * [new branch] gh/HDCharles/42/orig -> origin/gh/HDCharles/42/orig 2022-03-17T19:07:02.8390236Z * [new branch] gh/HDCharles/43/base -> origin/gh/HDCharles/43/base 2022-03-17T19:07:02.8391881Z * [new branch] gh/HDCharles/43/head -> origin/gh/HDCharles/43/head 2022-03-17T19:07:02.8393215Z * [new branch] gh/HDCharles/43/orig -> origin/gh/HDCharles/43/orig 2022-03-17T19:07:02.8395427Z * [new branch] gh/HDCharles/45/base -> origin/gh/HDCharles/45/base 2022-03-17T19:07:02.8396576Z * [new branch] gh/HDCharles/45/head -> origin/gh/HDCharles/45/head 2022-03-17T19:07:02.8398525Z * [new branch] gh/HDCharles/45/orig -> origin/gh/HDCharles/45/orig 2022-03-17T19:07:02.8399659Z * [new branch] gh/HDCharles/46/base -> origin/gh/HDCharles/46/base 2022-03-17T19:07:02.8400912Z * [new branch] gh/HDCharles/46/head -> origin/gh/HDCharles/46/head 2022-03-17T19:07:02.8402196Z * [new branch] gh/HDCharles/46/orig -> origin/gh/HDCharles/46/orig 2022-03-17T19:07:02.8403659Z * [new branch] gh/HDCharles/47/base -> origin/gh/HDCharles/47/base 2022-03-17T19:07:02.8405663Z * [new branch] gh/HDCharles/47/head -> origin/gh/HDCharles/47/head 2022-03-17T19:07:02.8406692Z * [new branch] gh/HDCharles/47/orig -> origin/gh/HDCharles/47/orig 2022-03-17T19:07:02.8408559Z * [new branch] gh/HDCharles/48/base -> origin/gh/HDCharles/48/base 2022-03-17T19:07:02.8409619Z * [new branch] gh/HDCharles/48/head -> origin/gh/HDCharles/48/head 2022-03-17T19:07:02.8411329Z * [new branch] gh/HDCharles/48/orig -> origin/gh/HDCharles/48/orig 2022-03-17T19:07:02.8413478Z * [new branch] gh/HDCharles/49/base -> origin/gh/HDCharles/49/base 2022-03-17T19:07:02.8414535Z * [new branch] gh/HDCharles/49/head -> origin/gh/HDCharles/49/head 2022-03-17T19:07:02.8416394Z * [new branch] gh/HDCharles/49/orig -> origin/gh/HDCharles/49/orig 2022-03-17T19:07:02.8417650Z * [new branch] gh/HDCharles/50/base -> origin/gh/HDCharles/50/base 2022-03-17T19:07:02.8418941Z * [new branch] gh/HDCharles/50/head -> origin/gh/HDCharles/50/head 2022-03-17T19:07:02.8420504Z * [new branch] gh/HDCharles/50/orig -> origin/gh/HDCharles/50/orig 2022-03-17T19:07:02.8432561Z * [new branch] gh/HDCharles/51/base -> origin/gh/HDCharles/51/base 2022-03-17T19:07:02.8433035Z * [new branch] gh/HDCharles/51/head -> origin/gh/HDCharles/51/head 2022-03-17T19:07:02.8433519Z * [new branch] gh/HDCharles/51/orig -> origin/gh/HDCharles/51/orig 2022-03-17T19:07:02.8433901Z * [new branch] gh/HDCharles/52/base -> origin/gh/HDCharles/52/base 2022-03-17T19:07:02.8435018Z * [new branch] gh/HDCharles/52/head -> origin/gh/HDCharles/52/head 2022-03-17T19:07:02.8435495Z * [new branch] gh/HDCharles/52/orig -> origin/gh/HDCharles/52/orig 2022-03-17T19:07:02.8435956Z * [new branch] gh/HDCharles/53/base -> origin/gh/HDCharles/53/base 2022-03-17T19:07:02.8436373Z * [new branch] gh/HDCharles/53/head -> origin/gh/HDCharles/53/head 2022-03-17T19:07:02.8436831Z * [new branch] gh/HDCharles/53/orig -> origin/gh/HDCharles/53/orig 2022-03-17T19:07:02.8437291Z * [new branch] gh/HDCharles/54/base -> origin/gh/HDCharles/54/base 2022-03-17T19:07:02.8437806Z * [new branch] gh/HDCharles/54/head -> origin/gh/HDCharles/54/head 2022-03-17T19:07:02.8438593Z * [new branch] gh/HDCharles/54/orig -> origin/gh/HDCharles/54/orig 2022-03-17T19:07:02.8441176Z * [new branch] gh/HDCharles/55/base -> origin/gh/HDCharles/55/base 2022-03-17T19:07:02.8442026Z * [new branch] gh/HDCharles/55/head -> origin/gh/HDCharles/55/head 2022-03-17T19:07:02.8443843Z * [new branch] gh/HDCharles/55/orig -> origin/gh/HDCharles/55/orig 2022-03-17T19:07:02.8445744Z * [new branch] gh/HDCharles/56/base -> origin/gh/HDCharles/56/base 2022-03-17T19:07:02.8447165Z * [new branch] gh/HDCharles/56/head -> origin/gh/HDCharles/56/head 2022-03-17T19:07:02.8448562Z * [new branch] gh/HDCharles/56/orig -> origin/gh/HDCharles/56/orig 2022-03-17T19:07:02.8451007Z * [new branch] gh/HDCharles/57/base -> origin/gh/HDCharles/57/base 2022-03-17T19:07:02.8452210Z * [new branch] gh/HDCharles/57/head -> origin/gh/HDCharles/57/head 2022-03-17T19:07:02.8454002Z * [new branch] gh/HDCharles/57/orig -> origin/gh/HDCharles/57/orig 2022-03-17T19:07:02.8455593Z * [new branch] gh/HDCharles/58/base -> origin/gh/HDCharles/58/base 2022-03-17T19:07:02.8456818Z * [new branch] gh/HDCharles/58/head -> origin/gh/HDCharles/58/head 2022-03-17T19:07:02.8459110Z * [new branch] gh/HDCharles/58/orig -> origin/gh/HDCharles/58/orig 2022-03-17T19:07:02.8460773Z * [new branch] gh/HDCharles/59/base -> origin/gh/HDCharles/59/base 2022-03-17T19:07:02.8462172Z * [new branch] gh/HDCharles/59/head -> origin/gh/HDCharles/59/head 2022-03-17T19:07:02.8464212Z * [new branch] gh/HDCharles/59/orig -> origin/gh/HDCharles/59/orig 2022-03-17T19:07:02.8465626Z * [new branch] gh/HDCharles/60/base -> origin/gh/HDCharles/60/base 2022-03-17T19:07:02.8467180Z * [new branch] gh/HDCharles/60/head -> origin/gh/HDCharles/60/head 2022-03-17T19:07:02.8468612Z * [new branch] gh/HDCharles/60/orig -> origin/gh/HDCharles/60/orig 2022-03-17T19:07:02.8471524Z * [new branch] gh/HDCharles/61/base -> origin/gh/HDCharles/61/base 2022-03-17T19:07:02.8473620Z * [new branch] gh/HDCharles/61/head -> origin/gh/HDCharles/61/head 2022-03-17T19:07:02.8474721Z * [new branch] gh/HDCharles/61/orig -> origin/gh/HDCharles/61/orig 2022-03-17T19:07:02.8476660Z * [new branch] gh/HDCharles/9/base -> origin/gh/HDCharles/9/base 2022-03-17T19:07:02.8478097Z * [new branch] gh/HDCharles/9/head -> origin/gh/HDCharles/9/head 2022-03-17T19:07:02.8479335Z * [new branch] gh/HDCharles/9/orig -> origin/gh/HDCharles/9/orig 2022-03-17T19:07:02.8481446Z * [new branch] gh/HaomiaoLiu730/1/base -> origin/gh/HaomiaoLiu730/1/base 2022-03-17T19:07:02.8482442Z * [new branch] gh/HaomiaoLiu730/1/head -> origin/gh/HaomiaoLiu730/1/head 2022-03-17T19:07:02.8484855Z * [new branch] gh/IvanKobzarev/1/base -> origin/gh/IvanKobzarev/1/base 2022-03-17T19:07:02.8485899Z * [new branch] gh/IvanKobzarev/1/head -> origin/gh/IvanKobzarev/1/head 2022-03-17T19:07:02.8487790Z * [new branch] gh/IvanKobzarev/19/base -> origin/gh/IvanKobzarev/19/base 2022-03-17T19:07:02.8488970Z * [new branch] gh/IvanKobzarev/19/head -> origin/gh/IvanKobzarev/19/head 2022-03-17T19:07:02.8490229Z * [new branch] gh/IvanKobzarev/19/orig -> origin/gh/IvanKobzarev/19/orig 2022-03-17T19:07:02.8491654Z * [new branch] gh/IvanKobzarev/2/base -> origin/gh/IvanKobzarev/2/base 2022-03-17T19:07:02.8493094Z * [new branch] gh/IvanKobzarev/2/head -> origin/gh/IvanKobzarev/2/head 2022-03-17T19:07:02.8494875Z * [new branch] gh/IvanKobzarev/26/base -> origin/gh/IvanKobzarev/26/base 2022-03-17T19:07:02.8498113Z * [new branch] gh/IvanKobzarev/26/head -> origin/gh/IvanKobzarev/26/head 2022-03-17T19:07:02.8498689Z * [new branch] gh/IvanKobzarev/26/orig -> origin/gh/IvanKobzarev/26/orig 2022-03-17T19:07:02.8499488Z * [new branch] gh/IvanKobzarev/29/base -> origin/gh/IvanKobzarev/29/base 2022-03-17T19:07:02.8501074Z * [new branch] gh/IvanKobzarev/29/head -> origin/gh/IvanKobzarev/29/head 2022-03-17T19:07:02.8502328Z * [new branch] gh/IvanKobzarev/29/orig -> origin/gh/IvanKobzarev/29/orig 2022-03-17T19:07:02.8504653Z * [new branch] gh/IvanKobzarev/44/base -> origin/gh/IvanKobzarev/44/base 2022-03-17T19:07:02.8505421Z * [new branch] gh/IvanKobzarev/44/head -> origin/gh/IvanKobzarev/44/head 2022-03-17T19:07:02.8507060Z * [new branch] gh/IvanKobzarev/44/orig -> origin/gh/IvanKobzarev/44/orig 2022-03-17T19:07:02.8509045Z * [new branch] gh/IvanKobzarev/47/base -> origin/gh/IvanKobzarev/47/base 2022-03-17T19:07:02.8510066Z * [new branch] gh/IvanKobzarev/47/head -> origin/gh/IvanKobzarev/47/head 2022-03-17T19:07:02.8511422Z * [new branch] gh/IvanKobzarev/47/orig -> origin/gh/IvanKobzarev/47/orig 2022-03-17T19:07:02.8513508Z * [new branch] gh/IvanKobzarev/60/base -> origin/gh/IvanKobzarev/60/base 2022-03-17T19:07:02.8514429Z * [new branch] gh/IvanKobzarev/60/head -> origin/gh/IvanKobzarev/60/head 2022-03-17T19:07:02.8515850Z * [new branch] gh/IvanKobzarev/60/orig -> origin/gh/IvanKobzarev/60/orig 2022-03-17T19:07:02.8533624Z * [new branch] gh/IvanKobzarev/61/base -> origin/gh/IvanKobzarev/61/base 2022-03-17T19:07:02.8534307Z * [new branch] gh/IvanKobzarev/61/head -> origin/gh/IvanKobzarev/61/head 2022-03-17T19:07:02.8534925Z * [new branch] gh/IvanKobzarev/61/orig -> origin/gh/IvanKobzarev/61/orig 2022-03-17T19:07:02.8535570Z * [new branch] gh/IvanKobzarev/62/base -> origin/gh/IvanKobzarev/62/base 2022-03-17T19:07:02.8536143Z * [new branch] gh/IvanKobzarev/62/head -> origin/gh/IvanKobzarev/62/head 2022-03-17T19:07:02.8536788Z * [new branch] gh/IvanKobzarev/62/orig -> origin/gh/IvanKobzarev/62/orig 2022-03-17T19:07:02.8539341Z * [new branch] gh/IvanKobzarev/63/base -> origin/gh/IvanKobzarev/63/base 2022-03-17T19:07:02.8540012Z * [new branch] gh/IvanKobzarev/63/head -> origin/gh/IvanKobzarev/63/head 2022-03-17T19:07:02.8540915Z * [new branch] gh/IvanKobzarev/63/orig -> origin/gh/IvanKobzarev/63/orig 2022-03-17T19:07:02.8541462Z * [new branch] gh/IvanKobzarev/67/base -> origin/gh/IvanKobzarev/67/base 2022-03-17T19:07:02.8542060Z * [new branch] gh/IvanKobzarev/67/head -> origin/gh/IvanKobzarev/67/head 2022-03-17T19:07:02.8542657Z * [new branch] gh/IvanKobzarev/67/orig -> origin/gh/IvanKobzarev/67/orig 2022-03-17T19:07:02.8543176Z * [new branch] gh/IvanKobzarev/68/base -> origin/gh/IvanKobzarev/68/base 2022-03-17T19:07:02.8543739Z * [new branch] gh/IvanKobzarev/68/head -> origin/gh/IvanKobzarev/68/head 2022-03-17T19:07:02.8544274Z * [new branch] gh/IvanKobzarev/68/orig -> origin/gh/IvanKobzarev/68/orig 2022-03-17T19:07:02.8544941Z * [new branch] gh/IvanKobzarev/74/base -> origin/gh/IvanKobzarev/74/base 2022-03-17T19:07:02.8545532Z * [new branch] gh/IvanKobzarev/74/head -> origin/gh/IvanKobzarev/74/head 2022-03-17T19:07:02.8546063Z * [new branch] gh/IvanKobzarev/74/orig -> origin/gh/IvanKobzarev/74/orig 2022-03-17T19:07:02.8546661Z * [new branch] gh/IvanKobzarev/83/base -> origin/gh/IvanKobzarev/83/base 2022-03-17T19:07:02.8547260Z * [new branch] gh/IvanKobzarev/83/head -> origin/gh/IvanKobzarev/83/head 2022-03-17T19:07:02.8547823Z * [new branch] gh/IvanKobzarev/83/orig -> origin/gh/IvanKobzarev/83/orig 2022-03-17T19:07:02.8549000Z * [new branch] gh/IvanKobzarev/85/base -> origin/gh/IvanKobzarev/85/base 2022-03-17T19:07:02.8550233Z * [new branch] gh/IvanKobzarev/85/head -> origin/gh/IvanKobzarev/85/head 2022-03-17T19:07:02.8551633Z * [new branch] gh/IvanKobzarev/85/orig -> origin/gh/IvanKobzarev/85/orig 2022-03-17T19:07:02.8554094Z * [new branch] gh/Lezcano/16/base -> origin/gh/Lezcano/16/base 2022-03-17T19:07:02.8555224Z * [new branch] gh/Lezcano/16/head -> origin/gh/Lezcano/16/head 2022-03-17T19:07:02.8557536Z * [new branch] gh/Lezcano/16/orig -> origin/gh/Lezcano/16/orig 2022-03-17T19:07:02.8559010Z * [new branch] gh/Lezcano/25/base -> origin/gh/Lezcano/25/base 2022-03-17T19:07:02.8561103Z * [new branch] gh/Lezcano/25/head -> origin/gh/Lezcano/25/head 2022-03-17T19:07:02.8562102Z * [new branch] gh/Lezcano/25/orig -> origin/gh/Lezcano/25/orig 2022-03-17T19:07:02.8565031Z * [new branch] gh/Lezcano/28/base -> origin/gh/Lezcano/28/base 2022-03-17T19:07:02.8566255Z * [new branch] gh/Lezcano/28/head -> origin/gh/Lezcano/28/head 2022-03-17T19:07:02.8567817Z * [new branch] gh/Lezcano/28/orig -> origin/gh/Lezcano/28/orig 2022-03-17T19:07:02.8569862Z * [new branch] gh/Lezcano/46/base -> origin/gh/Lezcano/46/base 2022-03-17T19:07:02.8571030Z * [new branch] gh/Lezcano/46/head -> origin/gh/Lezcano/46/head 2022-03-17T19:07:02.8572500Z * [new branch] gh/Lezcano/46/orig -> origin/gh/Lezcano/46/orig 2022-03-17T19:07:02.8578610Z * [new branch] gh/Lezcano/47/base -> origin/gh/Lezcano/47/base 2022-03-17T19:07:02.8581721Z * [new branch] gh/Lezcano/47/head -> origin/gh/Lezcano/47/head 2022-03-17T19:07:02.8583966Z * [new branch] gh/Lezcano/47/orig -> origin/gh/Lezcano/47/orig 2022-03-17T19:07:02.8586193Z * [new branch] gh/Lezcano/48/base -> origin/gh/Lezcano/48/base 2022-03-17T19:07:02.8587412Z * [new branch] gh/Lezcano/48/head -> origin/gh/Lezcano/48/head 2022-03-17T19:07:02.8589166Z * [new branch] gh/Lezcano/48/orig -> origin/gh/Lezcano/48/orig 2022-03-17T19:07:02.8591124Z * [new branch] gh/Lezcano/49/base -> origin/gh/Lezcano/49/base 2022-03-17T19:07:02.8592512Z * [new branch] gh/Lezcano/49/head -> origin/gh/Lezcano/49/head 2022-03-17T19:07:02.8594012Z * [new branch] gh/Lezcano/49/orig -> origin/gh/Lezcano/49/orig 2022-03-17T19:07:02.8595474Z * [new branch] gh/Lezcano/50/base -> origin/gh/Lezcano/50/base 2022-03-17T19:07:02.8596784Z * [new branch] gh/Lezcano/50/head -> origin/gh/Lezcano/50/head 2022-03-17T19:07:02.8598076Z * [new branch] gh/Lezcano/50/orig -> origin/gh/Lezcano/50/orig 2022-03-17T19:07:02.8600123Z * [new branch] gh/Lezcano/51/base -> origin/gh/Lezcano/51/base 2022-03-17T19:07:02.8601395Z * [new branch] gh/Lezcano/51/head -> origin/gh/Lezcano/51/head 2022-03-17T19:07:02.8602727Z * [new branch] gh/Lezcano/51/orig -> origin/gh/Lezcano/51/orig 2022-03-17T19:07:02.8605991Z * [new branch] gh/Lezcano/52/base -> origin/gh/Lezcano/52/base 2022-03-17T19:07:02.8607968Z * [new branch] gh/Lezcano/52/head -> origin/gh/Lezcano/52/head 2022-03-17T19:07:02.8609191Z * [new branch] gh/Lezcano/52/orig -> origin/gh/Lezcano/52/orig 2022-03-17T19:07:02.8611323Z * [new branch] gh/Lezcano/53/base -> origin/gh/Lezcano/53/base 2022-03-17T19:07:02.8615030Z * [new branch] gh/Lezcano/53/head -> origin/gh/Lezcano/53/head 2022-03-17T19:07:02.8615577Z * [new branch] gh/Lezcano/53/orig -> origin/gh/Lezcano/53/orig 2022-03-17T19:07:02.8616386Z * [new branch] gh/Lezcano/54/base -> origin/gh/Lezcano/54/base 2022-03-17T19:07:02.8746639Z * [new branch] gh/Lezcano/54/head -> origin/gh/Lezcano/54/head 2022-03-17T19:07:02.8747154Z * [new branch] gh/Lezcano/54/orig -> origin/gh/Lezcano/54/orig 2022-03-17T19:07:02.8747630Z * [new branch] gh/Lilyjjo/16/base -> origin/gh/Lilyjjo/16/base 2022-03-17T19:07:02.8748046Z * [new branch] gh/Lilyjjo/16/orig -> origin/gh/Lilyjjo/16/orig 2022-03-17T19:07:02.8748518Z * [new branch] gh/Lilyjjo/19/base -> origin/gh/Lilyjjo/19/base 2022-03-17T19:07:02.8748945Z * [new branch] gh/Lilyjjo/19/orig -> origin/gh/Lilyjjo/19/orig 2022-03-17T19:07:02.8749519Z * [new branch] gh/Lilyjjo/2/base -> origin/gh/Lilyjjo/2/base 2022-03-17T19:07:02.8750032Z * [new branch] gh/Lilyjjo/2/orig -> origin/gh/Lilyjjo/2/orig 2022-03-17T19:07:02.8750477Z * [new branch] gh/Lilyjjo/20/base -> origin/gh/Lilyjjo/20/base 2022-03-17T19:07:02.8750912Z * [new branch] gh/Lilyjjo/20/orig -> origin/gh/Lilyjjo/20/orig 2022-03-17T19:07:02.8751354Z * [new branch] gh/Lilyjjo/21/base -> origin/gh/Lilyjjo/21/base 2022-03-17T19:07:02.8751772Z * [new branch] gh/Lilyjjo/21/orig -> origin/gh/Lilyjjo/21/orig 2022-03-17T19:07:02.8752233Z * [new branch] gh/Lilyjjo/22/base -> origin/gh/Lilyjjo/22/base 2022-03-17T19:07:02.8752649Z * [new branch] gh/Lilyjjo/22/orig -> origin/gh/Lilyjjo/22/orig 2022-03-17T19:07:02.8753094Z * [new branch] gh/Lilyjjo/72/base -> origin/gh/Lilyjjo/72/base 2022-03-17T19:07:02.8753522Z * [new branch] gh/Lilyjjo/72/head -> origin/gh/Lilyjjo/72/head 2022-03-17T19:07:02.8754076Z * [new branch] gh/Lilyjjo/73/base -> origin/gh/Lilyjjo/73/base 2022-03-17T19:07:02.8754518Z * [new branch] gh/Lilyjjo/73/orig -> origin/gh/Lilyjjo/73/orig 2022-03-17T19:07:02.8754979Z * [new branch] gh/NicolasHug/1/base -> origin/gh/NicolasHug/1/base 2022-03-17T19:07:02.8755466Z * [new branch] gh/NicolasHug/1/head -> origin/gh/NicolasHug/1/head 2022-03-17T19:07:02.8755957Z * [new branch] gh/NicolasHug/1/orig -> origin/gh/NicolasHug/1/orig 2022-03-17T19:07:02.8756415Z * [new branch] gh/NicolasHug/2/base -> origin/gh/NicolasHug/2/base 2022-03-17T19:07:02.8756855Z * [new branch] gh/NicolasHug/2/head -> origin/gh/NicolasHug/2/head 2022-03-17T19:07:02.8757311Z * [new branch] gh/NicolasHug/2/orig -> origin/gh/NicolasHug/2/orig 2022-03-17T19:07:02.8757767Z * [new branch] gh/PaliC/1/base -> origin/gh/PaliC/1/base 2022-03-17T19:07:02.8758278Z * [new branch] gh/PaliC/1/orig -> origin/gh/PaliC/1/orig 2022-03-17T19:07:02.8758759Z * [new branch] gh/PaliC/105/base -> origin/gh/PaliC/105/base 2022-03-17T19:07:02.8759164Z * [new branch] gh/PaliC/105/head -> origin/gh/PaliC/105/head 2022-03-17T19:07:02.8759602Z * [new branch] gh/PaliC/106/base -> origin/gh/PaliC/106/base 2022-03-17T19:07:02.8760047Z * [new branch] gh/PaliC/106/orig -> origin/gh/PaliC/106/orig 2022-03-17T19:07:02.8760478Z * [new branch] gh/PaliC/110/base -> origin/gh/PaliC/110/base 2022-03-17T19:07:02.8760919Z * [new branch] gh/PaliC/110/head -> origin/gh/PaliC/110/head 2022-03-17T19:07:02.8761350Z * [new branch] gh/PaliC/111/base -> origin/gh/PaliC/111/base 2022-03-17T19:07:02.8761789Z * [new branch] gh/PaliC/111/orig -> origin/gh/PaliC/111/orig 2022-03-17T19:07:02.8762227Z * [new branch] gh/PaliC/112/base -> origin/gh/PaliC/112/base 2022-03-17T19:07:02.8762652Z * [new branch] gh/PaliC/112/orig -> origin/gh/PaliC/112/orig 2022-03-17T19:07:02.8763087Z * [new branch] gh/PaliC/113/base -> origin/gh/PaliC/113/base 2022-03-17T19:07:02.8763515Z * [new branch] gh/PaliC/113/orig -> origin/gh/PaliC/113/orig 2022-03-17T19:07:02.8763946Z * [new branch] gh/PaliC/114/base -> origin/gh/PaliC/114/base 2022-03-17T19:07:02.8764376Z * [new branch] gh/PaliC/114/head -> origin/gh/PaliC/114/head 2022-03-17T19:07:02.8764814Z * [new branch] gh/PaliC/115/base -> origin/gh/PaliC/115/base 2022-03-17T19:07:02.8765273Z * [new branch] gh/PaliC/115/orig -> origin/gh/PaliC/115/orig 2022-03-17T19:07:02.8765742Z * [new branch] gh/PaliC/116/base -> origin/gh/PaliC/116/base 2022-03-17T19:07:02.8766191Z * [new branch] gh/PaliC/116/orig -> origin/gh/PaliC/116/orig 2022-03-17T19:07:02.8766630Z * [new branch] gh/PaliC/117/base -> origin/gh/PaliC/117/base 2022-03-17T19:07:02.8767052Z * [new branch] gh/PaliC/117/orig -> origin/gh/PaliC/117/orig 2022-03-17T19:07:02.8767479Z * [new branch] gh/PaliC/120/base -> origin/gh/PaliC/120/base 2022-03-17T19:07:02.8767909Z * [new branch] gh/PaliC/120/head -> origin/gh/PaliC/120/head 2022-03-17T19:07:02.8768332Z * [new branch] gh/PaliC/120/orig -> origin/gh/PaliC/120/orig 2022-03-17T19:07:02.8768773Z * [new branch] gh/PaliC/121/base -> origin/gh/PaliC/121/base 2022-03-17T19:07:02.8769207Z * [new branch] gh/PaliC/121/head -> origin/gh/PaliC/121/head 2022-03-17T19:07:02.8769666Z * [new branch] gh/PaliC/121/orig -> origin/gh/PaliC/121/orig 2022-03-17T19:07:02.8770097Z * [new branch] gh/PaliC/122/base -> origin/gh/PaliC/122/base 2022-03-17T19:07:02.8770589Z * [new branch] gh/PaliC/122/head -> origin/gh/PaliC/122/head 2022-03-17T19:07:02.8771034Z * [new branch] gh/PaliC/122/orig -> origin/gh/PaliC/122/orig 2022-03-17T19:07:02.8771492Z * [new branch] gh/PaliC/123/base -> origin/gh/PaliC/123/base 2022-03-17T19:07:02.8771900Z * [new branch] gh/PaliC/123/head -> origin/gh/PaliC/123/head 2022-03-17T19:07:02.8772348Z * [new branch] gh/PaliC/123/orig -> origin/gh/PaliC/123/orig 2022-03-17T19:07:02.8772780Z * [new branch] gh/PaliC/124/base -> origin/gh/PaliC/124/base 2022-03-17T19:07:02.8773226Z * [new branch] gh/PaliC/124/head -> origin/gh/PaliC/124/head 2022-03-17T19:07:02.8773657Z * [new branch] gh/PaliC/124/orig -> origin/gh/PaliC/124/orig 2022-03-17T19:07:02.8774110Z * [new branch] gh/PaliC/126/base -> origin/gh/PaliC/126/base 2022-03-17T19:07:02.8774564Z * [new branch] gh/PaliC/126/head -> origin/gh/PaliC/126/head 2022-03-17T19:07:02.8774999Z * [new branch] gh/PaliC/126/orig -> origin/gh/PaliC/126/orig 2022-03-17T19:07:02.8775441Z * [new branch] gh/PaliC/127/base -> origin/gh/PaliC/127/base 2022-03-17T19:07:02.8775876Z * [new branch] gh/PaliC/127/head -> origin/gh/PaliC/127/head 2022-03-17T19:07:02.8776308Z * [new branch] gh/PaliC/127/orig -> origin/gh/PaliC/127/orig 2022-03-17T19:07:02.8776758Z * [new branch] gh/PaliC/128/base -> origin/gh/PaliC/128/base 2022-03-17T19:07:02.8777230Z * [new branch] gh/PaliC/128/head -> origin/gh/PaliC/128/head 2022-03-17T19:07:02.8777638Z * [new branch] gh/PaliC/128/orig -> origin/gh/PaliC/128/orig 2022-03-17T19:07:02.8778086Z * [new branch] gh/PaliC/129/base -> origin/gh/PaliC/129/base 2022-03-17T19:07:02.8778527Z * [new branch] gh/PaliC/129/head -> origin/gh/PaliC/129/head 2022-03-17T19:07:02.8778985Z * [new branch] gh/PaliC/129/orig -> origin/gh/PaliC/129/orig 2022-03-17T19:07:02.8779418Z * [new branch] gh/PaliC/130/base -> origin/gh/PaliC/130/base 2022-03-17T19:07:02.8779860Z * [new branch] gh/PaliC/130/head -> origin/gh/PaliC/130/head 2022-03-17T19:07:02.8780561Z * [new branch] gh/PaliC/130/orig -> origin/gh/PaliC/130/orig 2022-03-17T19:07:02.8781015Z * [new branch] gh/PaliC/131/base -> origin/gh/PaliC/131/base 2022-03-17T19:07:02.8781466Z * [new branch] gh/PaliC/131/head -> origin/gh/PaliC/131/head 2022-03-17T19:07:02.8781906Z * [new branch] gh/PaliC/131/orig -> origin/gh/PaliC/131/orig 2022-03-17T19:07:02.8782433Z * [new branch] gh/PaliC/132/base -> origin/gh/PaliC/132/base 2022-03-17T19:07:02.8782907Z * [new branch] gh/PaliC/132/head -> origin/gh/PaliC/132/head 2022-03-17T19:07:02.8783350Z * [new branch] gh/PaliC/132/orig -> origin/gh/PaliC/132/orig 2022-03-17T19:07:02.8783817Z * [new branch] gh/PaliC/133/base -> origin/gh/PaliC/133/base 2022-03-17T19:07:02.8784228Z * [new branch] gh/PaliC/133/head -> origin/gh/PaliC/133/head 2022-03-17T19:07:02.8784678Z * [new branch] gh/PaliC/133/orig -> origin/gh/PaliC/133/orig 2022-03-17T19:07:02.8785124Z * [new branch] gh/PaliC/134/base -> origin/gh/PaliC/134/base 2022-03-17T19:07:02.8785557Z * [new branch] gh/PaliC/134/head -> origin/gh/PaliC/134/head 2022-03-17T19:07:02.8786004Z * [new branch] gh/PaliC/134/orig -> origin/gh/PaliC/134/orig 2022-03-17T19:07:02.8786456Z * [new branch] gh/PaliC/135/base -> origin/gh/PaliC/135/base 2022-03-17T19:07:02.8786969Z * [new branch] gh/PaliC/135/head -> origin/gh/PaliC/135/head 2022-03-17T19:07:02.8787418Z * [new branch] gh/PaliC/135/orig -> origin/gh/PaliC/135/orig 2022-03-17T19:07:02.8787847Z * [new branch] gh/PaliC/136/base -> origin/gh/PaliC/136/base 2022-03-17T19:07:02.8788313Z * [new branch] gh/PaliC/136/head -> origin/gh/PaliC/136/head 2022-03-17T19:07:02.8788779Z * [new branch] gh/PaliC/136/orig -> origin/gh/PaliC/136/orig 2022-03-17T19:07:02.8789191Z * [new branch] gh/PaliC/137/base -> origin/gh/PaliC/137/base 2022-03-17T19:07:02.8789668Z * [new branch] gh/PaliC/137/head -> origin/gh/PaliC/137/head 2022-03-17T19:07:02.8790117Z * [new branch] gh/PaliC/137/orig -> origin/gh/PaliC/137/orig 2022-03-17T19:07:02.8790524Z * [new branch] gh/PaliC/138/base -> origin/gh/PaliC/138/base 2022-03-17T19:07:02.8791025Z * [new branch] gh/PaliC/138/head -> origin/gh/PaliC/138/head 2022-03-17T19:07:02.8791460Z * [new branch] gh/PaliC/138/orig -> origin/gh/PaliC/138/orig 2022-03-17T19:07:02.8791937Z * [new branch] gh/PaliC/139/base -> origin/gh/PaliC/139/base 2022-03-17T19:07:02.8792355Z * [new branch] gh/PaliC/139/head -> origin/gh/PaliC/139/head 2022-03-17T19:07:02.8792799Z * [new branch] gh/PaliC/139/orig -> origin/gh/PaliC/139/orig 2022-03-17T19:07:02.8793248Z * [new branch] gh/PaliC/140/base -> origin/gh/PaliC/140/base 2022-03-17T19:07:02.8793664Z * [new branch] gh/PaliC/140/head -> origin/gh/PaliC/140/head 2022-03-17T19:07:02.8794105Z * [new branch] gh/PaliC/140/orig -> origin/gh/PaliC/140/orig 2022-03-17T19:07:02.8794553Z * [new branch] gh/PaliC/142/base -> origin/gh/PaliC/142/base 2022-03-17T19:07:02.8794984Z * [new branch] gh/PaliC/142/head -> origin/gh/PaliC/142/head 2022-03-17T19:07:02.8795427Z * [new branch] gh/PaliC/142/orig -> origin/gh/PaliC/142/orig 2022-03-17T19:07:02.8795859Z * [new branch] gh/PaliC/143/base -> origin/gh/PaliC/143/base 2022-03-17T19:07:02.8796322Z * [new branch] gh/PaliC/143/head -> origin/gh/PaliC/143/head 2022-03-17T19:07:02.8796765Z * [new branch] gh/PaliC/143/orig -> origin/gh/PaliC/143/orig 2022-03-17T19:07:02.8797199Z * [new branch] gh/PaliC/144/base -> origin/gh/PaliC/144/base 2022-03-17T19:07:02.8797650Z * [new branch] gh/PaliC/144/head -> origin/gh/PaliC/144/head 2022-03-17T19:07:02.8798103Z * [new branch] gh/PaliC/145/base -> origin/gh/PaliC/145/base 2022-03-17T19:07:02.8798545Z * [new branch] gh/PaliC/145/head -> origin/gh/PaliC/145/head 2022-03-17T19:07:02.8799274Z * [new branch] gh/PaliC/145/orig -> origin/gh/PaliC/145/orig 2022-03-17T19:07:02.8801285Z * [new branch] gh/PaliC/146/base -> origin/gh/PaliC/146/base 2022-03-17T19:07:02.8802492Z * [new branch] gh/PaliC/146/head -> origin/gh/PaliC/146/head 2022-03-17T19:07:02.8803785Z * [new branch] gh/PaliC/146/orig -> origin/gh/PaliC/146/orig 2022-03-17T19:07:02.8806056Z * [new branch] gh/PaliC/147/base -> origin/gh/PaliC/147/base 2022-03-17T19:07:02.8807020Z * [new branch] gh/PaliC/147/head -> origin/gh/PaliC/147/head 2022-03-17T19:07:02.8808378Z * [new branch] gh/PaliC/147/orig -> origin/gh/PaliC/147/orig 2022-03-17T19:07:02.8810609Z * [new branch] gh/PaliC/148/base -> origin/gh/PaliC/148/base 2022-03-17T19:07:02.8811789Z * [new branch] gh/PaliC/148/head -> origin/gh/PaliC/148/head 2022-03-17T19:07:02.8813752Z * [new branch] gh/PaliC/148/orig -> origin/gh/PaliC/148/orig 2022-03-17T19:07:02.8814949Z * [new branch] gh/PaliC/2/base -> origin/gh/PaliC/2/base 2022-03-17T19:07:02.8816282Z * [new branch] gh/PaliC/2/orig -> origin/gh/PaliC/2/orig 2022-03-17T19:07:02.8818454Z * [new branch] gh/PaliC/42/base -> origin/gh/PaliC/42/base 2022-03-17T19:07:02.8819638Z * [new branch] gh/PaliC/42/orig -> origin/gh/PaliC/42/orig 2022-03-17T19:07:02.8822140Z * [new branch] gh/PaliC/5/base -> origin/gh/PaliC/5/base 2022-03-17T19:07:02.8823073Z * [new branch] gh/PaliC/5/orig -> origin/gh/PaliC/5/orig 2022-03-17T19:07:02.8825273Z * [new branch] gh/PaliC/56/base -> origin/gh/PaliC/56/base 2022-03-17T19:07:02.8826224Z * [new branch] gh/PaliC/56/head -> origin/gh/PaliC/56/head 2022-03-17T19:07:02.8827580Z * [new branch] gh/PaliC/56/orig -> origin/gh/PaliC/56/orig 2022-03-17T19:07:02.8829586Z * [new branch] gh/PaliC/60/base -> origin/gh/PaliC/60/base 2022-03-17T19:07:02.8830624Z * [new branch] gh/PaliC/60/orig -> origin/gh/PaliC/60/orig 2022-03-17T19:07:02.8832745Z * [new branch] gh/PaliC/72/base -> origin/gh/PaliC/72/base 2022-03-17T19:07:02.8834067Z * [new branch] gh/PaliC/94/base -> origin/gh/PaliC/94/base 2022-03-17T19:07:02.8836607Z * [new branch] gh/SS-JIA/21/base -> origin/gh/SS-JIA/21/base 2022-03-17T19:07:02.8837562Z * [new branch] gh/SS-JIA/21/head -> origin/gh/SS-JIA/21/head 2022-03-17T19:07:02.8838842Z * [new branch] gh/SS-JIA/21/orig -> origin/gh/SS-JIA/21/orig 2022-03-17T19:07:02.8841059Z * [new branch] gh/SS-JIA/23/base -> origin/gh/SS-JIA/23/base 2022-03-17T19:07:02.8842010Z * [new branch] gh/SS-JIA/23/head -> origin/gh/SS-JIA/23/head 2022-03-17T19:07:02.8843269Z * [new branch] gh/SS-JIA/23/orig -> origin/gh/SS-JIA/23/orig 2022-03-17T19:07:02.8845471Z * [new branch] gh/SS-JIA/24/base -> origin/gh/SS-JIA/24/base 2022-03-17T19:07:02.8846454Z * [new branch] gh/SS-JIA/24/head -> origin/gh/SS-JIA/24/head 2022-03-17T19:07:02.8847725Z * [new branch] gh/SS-JIA/24/orig -> origin/gh/SS-JIA/24/orig 2022-03-17T19:07:02.8849954Z * [new branch] gh/SS-JIA/25/base -> origin/gh/SS-JIA/25/base 2022-03-17T19:07:02.8850908Z * [new branch] gh/SS-JIA/25/head -> origin/gh/SS-JIA/25/head 2022-03-17T19:07:02.8852189Z * [new branch] gh/SS-JIA/25/orig -> origin/gh/SS-JIA/25/orig 2022-03-17T19:07:02.8854574Z * [new branch] gh/SS-JIA/26/base -> origin/gh/SS-JIA/26/base 2022-03-17T19:07:02.8856138Z * [new branch] gh/SS-JIA/26/head -> origin/gh/SS-JIA/26/head 2022-03-17T19:07:02.8857345Z * [new branch] gh/SS-JIA/26/orig -> origin/gh/SS-JIA/26/orig 2022-03-17T19:07:02.8859285Z * [new branch] gh/SS-JIA/29/base -> origin/gh/SS-JIA/29/base 2022-03-17T19:07:02.8860673Z * [new branch] gh/SS-JIA/29/head -> origin/gh/SS-JIA/29/head 2022-03-17T19:07:02.8862015Z * [new branch] gh/SS-JIA/29/orig -> origin/gh/SS-JIA/29/orig 2022-03-17T19:07:02.8863927Z * [new branch] gh/SS-JIA/30/base -> origin/gh/SS-JIA/30/base 2022-03-17T19:07:02.8865140Z * [new branch] gh/SS-JIA/30/orig -> origin/gh/SS-JIA/30/orig 2022-03-17T19:07:02.8867275Z * [new branch] gh/SS-JIA/31/base -> origin/gh/SS-JIA/31/base 2022-03-17T19:07:02.8868226Z * [new branch] gh/SS-JIA/31/orig -> origin/gh/SS-JIA/31/orig 2022-03-17T19:07:02.8869999Z * [new branch] gh/SS-JIA/32/base -> origin/gh/SS-JIA/32/base 2022-03-17T19:07:02.8871245Z * [new branch] gh/SS-JIA/32/orig -> origin/gh/SS-JIA/32/orig 2022-03-17T19:07:02.8873509Z * [new branch] gh/SS-JIA/33/base -> origin/gh/SS-JIA/33/base 2022-03-17T19:07:02.8882921Z * [new branch] gh/SS-JIA/33/orig -> origin/gh/SS-JIA/33/orig 2022-03-17T19:07:02.8883384Z * [new branch] gh/SS-JIA/34/base -> origin/gh/SS-JIA/34/base 2022-03-17T19:07:02.8883813Z * [new branch] gh/SS-JIA/34/orig -> origin/gh/SS-JIA/34/orig 2022-03-17T19:07:02.8884263Z * [new branch] gh/SS-JIA/35/base -> origin/gh/SS-JIA/35/base 2022-03-17T19:07:02.8884698Z * [new branch] gh/SS-JIA/35/orig -> origin/gh/SS-JIA/35/orig 2022-03-17T19:07:02.8885174Z * [new branch] gh/SS-JIA/36/base -> origin/gh/SS-JIA/36/base 2022-03-17T19:07:02.8885588Z * [new branch] gh/SS-JIA/36/orig -> origin/gh/SS-JIA/36/orig 2022-03-17T19:07:02.8886022Z * [new branch] gh/SS-JIA/40/base -> origin/gh/SS-JIA/40/base 2022-03-17T19:07:02.8886485Z * [new branch] gh/SS-JIA/40/head -> origin/gh/SS-JIA/40/head 2022-03-17T19:07:02.8887120Z * [new branch] gh/SS-JIA/40/orig -> origin/gh/SS-JIA/40/orig 2022-03-17T19:07:02.8888485Z * [new branch] gh/SS-JIA/41/base -> origin/gh/SS-JIA/41/base 2022-03-17T19:07:02.8890502Z * [new branch] gh/SS-JIA/41/orig -> origin/gh/SS-JIA/41/orig 2022-03-17T19:07:02.8892566Z * [new branch] gh/SS-JIA/42/base -> origin/gh/SS-JIA/42/base 2022-03-17T19:07:02.8893491Z * [new branch] gh/SS-JIA/42/orig -> origin/gh/SS-JIA/42/orig 2022-03-17T19:07:02.8895471Z * [new branch] gh/SS-JIA/43/base -> origin/gh/SS-JIA/43/base 2022-03-17T19:07:02.8896375Z * [new branch] gh/SS-JIA/43/head -> origin/gh/SS-JIA/43/head 2022-03-17T19:07:02.8897580Z * [new branch] gh/SS-JIA/43/orig -> origin/gh/SS-JIA/43/orig 2022-03-17T19:07:02.8899914Z * [new branch] gh/SciPioneer/1/base -> origin/gh/SciPioneer/1/base 2022-03-17T19:07:02.8901000Z * [new branch] gh/SciPioneer/1/head -> origin/gh/SciPioneer/1/head 2022-03-17T19:07:02.8903338Z * [new branch] gh/SciPioneer/10/base -> origin/gh/SciPioneer/10/base 2022-03-17T19:07:02.8904322Z * [new branch] gh/SciPioneer/10/head -> origin/gh/SciPioneer/10/head 2022-03-17T19:07:02.8905718Z * [new branch] gh/SciPioneer/11/base -> origin/gh/SciPioneer/11/base 2022-03-17T19:07:02.8907398Z * [new branch] gh/SciPioneer/11/head -> origin/gh/SciPioneer/11/head 2022-03-17T19:07:02.8909402Z * [new branch] gh/SciPioneer/147/base -> origin/gh/SciPioneer/147/base 2022-03-17T19:07:02.8910447Z * [new branch] gh/SciPioneer/147/head -> origin/gh/SciPioneer/147/head 2022-03-17T19:07:02.8911916Z * [new branch] gh/SciPioneer/147/orig -> origin/gh/SciPioneer/147/orig 2022-03-17T19:07:02.8913869Z * [new branch] gh/SciPioneer/2/base -> origin/gh/SciPioneer/2/base 2022-03-17T19:07:02.8914975Z * [new branch] gh/SciPioneer/2/head -> origin/gh/SciPioneer/2/head 2022-03-17T19:07:02.8917302Z * [new branch] gh/SciPioneer/22/base -> origin/gh/SciPioneer/22/base 2022-03-17T19:07:02.8918313Z * [new branch] gh/SciPioneer/22/head -> origin/gh/SciPioneer/22/head 2022-03-17T19:07:02.8920812Z * [new branch] gh/SciPioneer/3/base -> origin/gh/SciPioneer/3/base 2022-03-17T19:07:02.8921734Z * [new branch] gh/SciPioneer/3/head -> origin/gh/SciPioneer/3/head 2022-03-17T19:07:02.8923979Z * [new branch] gh/SciPioneer/36/base -> origin/gh/SciPioneer/36/base 2022-03-17T19:07:02.8924997Z * [new branch] gh/SciPioneer/36/head -> origin/gh/SciPioneer/36/head 2022-03-17T19:07:02.8927585Z * [new branch] gh/SciPioneer/36/orig -> origin/gh/SciPioneer/36/orig 2022-03-17T19:07:02.8928144Z * [new branch] gh/SciPioneer/37/base -> origin/gh/SciPioneer/37/base 2022-03-17T19:07:02.8929918Z * [new branch] gh/SciPioneer/37/head -> origin/gh/SciPioneer/37/head 2022-03-17T19:07:02.8931077Z * [new branch] gh/SciPioneer/37/orig -> origin/gh/SciPioneer/37/orig 2022-03-17T19:07:02.8932474Z * [new branch] gh/SciPioneer/4/base -> origin/gh/SciPioneer/4/base 2022-03-17T19:07:02.8933735Z * [new branch] gh/SciPioneer/4/head -> origin/gh/SciPioneer/4/head 2022-03-17T19:07:02.8935414Z * [new branch] gh/SciPioneer/5/base -> origin/gh/SciPioneer/5/base 2022-03-17T19:07:02.8936575Z * [new branch] gh/SciPioneer/5/head -> origin/gh/SciPioneer/5/head 2022-03-17T19:07:02.8938246Z * [new branch] gh/SciPioneer/6/base -> origin/gh/SciPioneer/6/base 2022-03-17T19:07:02.8939400Z * [new branch] gh/SciPioneer/6/head -> origin/gh/SciPioneer/6/head 2022-03-17T19:07:02.8943303Z * [new branch] gh/SciPioneer/7/base -> origin/gh/SciPioneer/7/base 2022-03-17T19:07:02.8944433Z * [new branch] gh/SciPioneer/7/head -> origin/gh/SciPioneer/7/head 2022-03-17T19:07:02.8946528Z * [new branch] gh/SciPioneer/70/base -> origin/gh/SciPioneer/70/base 2022-03-17T19:07:02.8947770Z * [new branch] gh/SciPioneer/70/head -> origin/gh/SciPioneer/70/head 2022-03-17T19:07:02.8949012Z * [new branch] gh/SciPioneer/70/orig -> origin/gh/SciPioneer/70/orig 2022-03-17T19:07:02.8951419Z * [new branch] gh/SciPioneer/8/base -> origin/gh/SciPioneer/8/base 2022-03-17T19:07:02.8952596Z * [new branch] gh/SciPioneer/8/head -> origin/gh/SciPioneer/8/head 2022-03-17T19:07:02.8954013Z * [new branch] gh/SciPioneer/9/base -> origin/gh/SciPioneer/9/base 2022-03-17T19:07:02.8955305Z * [new branch] gh/SciPioneer/9/head -> origin/gh/SciPioneer/9/head 2022-03-17T19:07:02.8957475Z * [new branch] gh/SherlockNoMad/1/head -> origin/gh/SherlockNoMad/1/head 2022-03-17T19:07:02.8959370Z * [new branch] gh/SiddhantRanade/1/base -> origin/gh/SiddhantRanade/1/base 2022-03-17T19:07:02.8960527Z * [new branch] gh/SiddhantRanade/1/head -> origin/gh/SiddhantRanade/1/head 2022-03-17T19:07:02.8962436Z * [new branch] gh/SiddhantRanade/1/orig -> origin/gh/SiddhantRanade/1/orig 2022-03-17T19:07:02.8965103Z * [new branch] gh/VitalyFedyunin/10/base -> origin/gh/VitalyFedyunin/10/base 2022-03-17T19:07:02.8966238Z * [new branch] gh/VitalyFedyunin/10/orig -> origin/gh/VitalyFedyunin/10/orig 2022-03-17T19:07:02.8968183Z * [new branch] gh/VitalyFedyunin/113/base -> origin/gh/VitalyFedyunin/113/base 2022-03-17T19:07:02.8969413Z * [new branch] gh/VitalyFedyunin/113/orig -> origin/gh/VitalyFedyunin/113/orig 2022-03-17T19:07:02.8971078Z * [new branch] gh/VitalyFedyunin/149/base -> origin/gh/VitalyFedyunin/149/base 2022-03-17T19:07:02.8972309Z * [new branch] gh/VitalyFedyunin/149/head -> origin/gh/VitalyFedyunin/149/head 2022-03-17T19:07:02.8973718Z * [new branch] gh/VitalyFedyunin/149/orig -> origin/gh/VitalyFedyunin/149/orig 2022-03-17T19:07:02.8975677Z * [new branch] gh/VitalyFedyunin/170/base -> origin/gh/VitalyFedyunin/170/base 2022-03-17T19:07:02.8976814Z * [new branch] gh/VitalyFedyunin/170/head -> origin/gh/VitalyFedyunin/170/head 2022-03-17T19:07:02.8978160Z * [new branch] gh/VitalyFedyunin/170/orig -> origin/gh/VitalyFedyunin/170/orig 2022-03-17T19:07:02.8980518Z * [new branch] gh/VitalyFedyunin/175/base -> origin/gh/VitalyFedyunin/175/base 2022-03-17T19:07:02.8981907Z * [new branch] gh/VitalyFedyunin/175/head -> origin/gh/VitalyFedyunin/175/head 2022-03-17T19:07:02.8983261Z * [new branch] gh/VitalyFedyunin/175/orig -> origin/gh/VitalyFedyunin/175/orig 2022-03-17T19:07:02.8985264Z * [new branch] gh/VitalyFedyunin/177/base -> origin/gh/VitalyFedyunin/177/base 2022-03-17T19:07:02.8986404Z * [new branch] gh/VitalyFedyunin/177/head -> origin/gh/VitalyFedyunin/177/head 2022-03-17T19:07:02.8987716Z * [new branch] gh/VitalyFedyunin/177/orig -> origin/gh/VitalyFedyunin/177/orig 2022-03-17T19:07:02.8989711Z * [new branch] gh/VitalyFedyunin/20/base -> origin/gh/VitalyFedyunin/20/base 2022-03-17T19:07:02.8990935Z * [new branch] gh/VitalyFedyunin/20/orig -> origin/gh/VitalyFedyunin/20/orig 2022-03-17T19:07:02.8992968Z * [new branch] gh/VitalyFedyunin/46/orig -> origin/gh/VitalyFedyunin/46/orig 2022-03-17T19:07:02.8995448Z * [new branch] gh/VitalyFedyunin/47/orig -> origin/gh/VitalyFedyunin/47/orig 2022-03-17T19:07:02.8996890Z * [new branch] gh/VitalyFedyunin/48/orig -> origin/gh/VitalyFedyunin/48/orig 2022-03-17T19:07:02.8998967Z * [new branch] gh/VitalyFedyunin/49/orig -> origin/gh/VitalyFedyunin/49/orig 2022-03-17T19:07:02.9000400Z * [new branch] gh/VitalyFedyunin/50/orig -> origin/gh/VitalyFedyunin/50/orig 2022-03-17T19:07:02.9002225Z * [new branch] gh/VitalyFedyunin/51/orig -> origin/gh/VitalyFedyunin/51/orig 2022-03-17T19:07:02.9003618Z * [new branch] gh/VitalyFedyunin/52/orig -> origin/gh/VitalyFedyunin/52/orig 2022-03-17T19:07:02.9005586Z * [new branch] gh/VitalyFedyunin/56/orig -> origin/gh/VitalyFedyunin/56/orig 2022-03-17T19:07:02.9007028Z * [new branch] gh/VitalyFedyunin/57/orig -> origin/gh/VitalyFedyunin/57/orig 2022-03-17T19:07:02.9008875Z * [new branch] gh/VitalyFedyunin/58/orig -> origin/gh/VitalyFedyunin/58/orig 2022-03-17T19:07:02.9010296Z * [new branch] gh/VitalyFedyunin/59/orig -> origin/gh/VitalyFedyunin/59/orig 2022-03-17T19:07:02.9012361Z * [new branch] gh/VitalyFedyunin/60/orig -> origin/gh/VitalyFedyunin/60/orig 2022-03-17T19:07:02.9013973Z * [new branch] gh/VitalyFedyunin/61/orig -> origin/gh/VitalyFedyunin/61/orig 2022-03-17T19:07:02.9015827Z * [new branch] gh/VitalyFedyunin/62/orig -> origin/gh/VitalyFedyunin/62/orig 2022-03-17T19:07:02.9017235Z * [new branch] gh/VitalyFedyunin/64/base -> origin/gh/VitalyFedyunin/64/base 2022-03-17T19:07:02.9019189Z * [new branch] gh/VitalyFedyunin/75/orig -> origin/gh/VitalyFedyunin/75/orig 2022-03-17T19:07:02.9020849Z * [new branch] gh/VitalyFedyunin/77/orig -> origin/gh/VitalyFedyunin/77/orig 2022-03-17T19:07:02.9022480Z * [new branch] gh/VitalyFedyunin/78/orig -> origin/gh/VitalyFedyunin/78/orig 2022-03-17T19:07:02.9024600Z * [new branch] gh/XiaobingSuper/9/base -> origin/gh/XiaobingSuper/9/base 2022-03-17T19:07:02.9025826Z * [new branch] gh/XiaobingSuper/9/orig -> origin/gh/XiaobingSuper/9/orig 2022-03-17T19:07:02.9028226Z * [new branch] gh/YazhiGao/1/base -> origin/gh/YazhiGao/1/base 2022-03-17T19:07:02.9029358Z * [new branch] gh/YazhiGao/1/orig -> origin/gh/YazhiGao/1/orig 2022-03-17T19:07:02.9031534Z * [new branch] gh/YesterdayDX/1/base -> origin/gh/YesterdayDX/1/base 2022-03-17T19:07:02.9032711Z * [new branch] gh/YesterdayDX/1/head -> origin/gh/YesterdayDX/1/head 2022-03-17T19:07:02.9034250Z * [new branch] gh/YesterdayDX/2/base -> origin/gh/YesterdayDX/2/base 2022-03-17T19:07:02.9035451Z * [new branch] gh/YesterdayDX/2/head -> origin/gh/YesterdayDX/2/head 2022-03-17T19:07:02.9036856Z * [new branch] gh/YesterdayDX/3/base -> origin/gh/YesterdayDX/3/base 2022-03-17T19:07:02.9038107Z * [new branch] gh/YesterdayDX/3/head -> origin/gh/YesterdayDX/3/head 2022-03-17T19:07:02.9040368Z * [new branch] gh/ZolotukhinM/1/base -> origin/gh/ZolotukhinM/1/base 2022-03-17T19:07:02.9041474Z * [new branch] gh/ZolotukhinM/1/head -> origin/gh/ZolotukhinM/1/head 2022-03-17T19:07:02.9043347Z * [new branch] gh/ZolotukhinM/1/orig -> origin/gh/ZolotukhinM/1/orig 2022-03-17T19:07:02.9044984Z * [new branch] gh/ZolotukhinM/192/base -> origin/gh/ZolotukhinM/192/base 2022-03-17T19:07:02.9046168Z * [new branch] gh/ZolotukhinM/192/head -> origin/gh/ZolotukhinM/192/head 2022-03-17T19:07:02.9048215Z * [new branch] gh/ZolotukhinM/22/base -> origin/gh/ZolotukhinM/22/base 2022-03-17T19:07:02.9049653Z * [new branch] gh/ZolotukhinM/242/head -> origin/gh/ZolotukhinM/242/head 2022-03-17T19:07:02.9051251Z * [new branch] gh/ZolotukhinM/242/orig -> origin/gh/ZolotukhinM/242/orig 2022-03-17T19:07:02.9053799Z * [new branch] gh/ZolotukhinM/354/base -> origin/gh/ZolotukhinM/354/base 2022-03-17T19:07:02.9054983Z * [new branch] gh/ZolotukhinM/354/head -> origin/gh/ZolotukhinM/354/head 2022-03-17T19:07:02.9056346Z * [new branch] gh/ZolotukhinM/354/orig -> origin/gh/ZolotukhinM/354/orig 2022-03-17T19:07:02.9058507Z * [new branch] gh/ZolotukhinM/355/base -> origin/gh/ZolotukhinM/355/base 2022-03-17T19:07:02.9059652Z * [new branch] gh/ZolotukhinM/355/head -> origin/gh/ZolotukhinM/355/head 2022-03-17T19:07:02.9060939Z * [new branch] gh/ZolotukhinM/355/orig -> origin/gh/ZolotukhinM/355/orig 2022-03-17T19:07:02.9063331Z * [new branch] gh/ZolotukhinM/404/base -> origin/gh/ZolotukhinM/404/base 2022-03-17T19:07:02.9064599Z * [new branch] gh/ZolotukhinM/404/head -> origin/gh/ZolotukhinM/404/head 2022-03-17T19:07:02.9065908Z * [new branch] gh/ZolotukhinM/404/orig -> origin/gh/ZolotukhinM/404/orig 2022-03-17T19:07:02.9067502Z * [new branch] gh/ZolotukhinM/405/base -> origin/gh/ZolotukhinM/405/base 2022-03-17T19:07:02.9068805Z * [new branch] gh/ZolotukhinM/405/head -> origin/gh/ZolotukhinM/405/head 2022-03-17T19:07:02.9070183Z * [new branch] gh/ZolotukhinM/405/orig -> origin/gh/ZolotukhinM/405/orig 2022-03-17T19:07:02.9072255Z * [new branch] gh/ZolotukhinM/443/base -> origin/gh/ZolotukhinM/443/base 2022-03-17T19:07:02.9073493Z * [new branch] gh/ZolotukhinM/443/head -> origin/gh/ZolotukhinM/443/head 2022-03-17T19:07:02.9074860Z * [new branch] gh/ZolotukhinM/443/orig -> origin/gh/ZolotukhinM/443/orig 2022-03-17T19:07:02.9076485Z * [new branch] gh/ZolotukhinM/444/base -> origin/gh/ZolotukhinM/444/base 2022-03-17T19:07:02.9077826Z * [new branch] gh/ZolotukhinM/444/head -> origin/gh/ZolotukhinM/444/head 2022-03-17T19:07:02.9079162Z * [new branch] gh/ZolotukhinM/444/orig -> origin/gh/ZolotukhinM/444/orig 2022-03-17T19:07:02.9081657Z * [new branch] gh/ZolotukhinM/448/base -> origin/gh/ZolotukhinM/448/base 2022-03-17T19:07:02.9082541Z * [new branch] gh/ZolotukhinM/448/head -> origin/gh/ZolotukhinM/448/head 2022-03-17T19:07:02.9083639Z * [new branch] gh/ZolotukhinM/448/orig -> origin/gh/ZolotukhinM/448/orig 2022-03-17T19:07:02.9085556Z * [new branch] gh/ZolotukhinM/449/base -> origin/gh/ZolotukhinM/449/base 2022-03-17T19:07:02.9086510Z * [new branch] gh/ZolotukhinM/449/head -> origin/gh/ZolotukhinM/449/head 2022-03-17T19:07:02.9087852Z * [new branch] gh/ZolotukhinM/449/orig -> origin/gh/ZolotukhinM/449/orig 2022-03-17T19:07:02.9089946Z * [new branch] gh/ZolotukhinM/454/base -> origin/gh/ZolotukhinM/454/base 2022-03-17T19:07:02.9091012Z * [new branch] gh/ZolotukhinM/454/head -> origin/gh/ZolotukhinM/454/head 2022-03-17T19:07:02.9092639Z * [new branch] gh/ZolotukhinM/454/orig -> origin/gh/ZolotukhinM/454/orig 2022-03-17T19:07:02.9094862Z * [new branch] gh/ZolotukhinM/464/base -> origin/gh/ZolotukhinM/464/base 2022-03-17T19:07:02.9095618Z * [new branch] gh/ZolotukhinM/464/head -> origin/gh/ZolotukhinM/464/head 2022-03-17T19:07:02.9097220Z * [new branch] gh/ZolotukhinM/464/orig -> origin/gh/ZolotukhinM/464/orig 2022-03-17T19:07:02.9099197Z * [new branch] gh/ZolotukhinM/468/base -> origin/gh/ZolotukhinM/468/base 2022-03-17T19:07:02.9100608Z * [new branch] gh/ZolotukhinM/468/head -> origin/gh/ZolotukhinM/468/head 2022-03-17T19:07:02.9102980Z * [new branch] gh/ZolotukhinM/468/orig -> origin/gh/ZolotukhinM/468/orig 2022-03-17T19:07:02.9104144Z * [new branch] gh/ZolotukhinM/469/base -> origin/gh/ZolotukhinM/469/base 2022-03-17T19:07:02.9105613Z * [new branch] gh/ZolotukhinM/469/head -> origin/gh/ZolotukhinM/469/head 2022-03-17T19:07:02.9106914Z * [new branch] gh/ZolotukhinM/469/orig -> origin/gh/ZolotukhinM/469/orig 2022-03-17T19:07:02.9108517Z * [new branch] gh/ZolotukhinM/470/base -> origin/gh/ZolotukhinM/470/base 2022-03-17T19:07:02.9109722Z * [new branch] gh/ZolotukhinM/470/head -> origin/gh/ZolotukhinM/470/head 2022-03-17T19:07:02.9111172Z * [new branch] gh/ZolotukhinM/470/orig -> origin/gh/ZolotukhinM/470/orig 2022-03-17T19:07:02.9112510Z * [new branch] gh/ZolotukhinM/471/base -> origin/gh/ZolotukhinM/471/base 2022-03-17T19:07:02.9113877Z * [new branch] gh/ZolotukhinM/471/head -> origin/gh/ZolotukhinM/471/head 2022-03-17T19:07:02.9115271Z * [new branch] gh/ZolotukhinM/471/orig -> origin/gh/ZolotukhinM/471/orig 2022-03-17T19:07:02.9117062Z * [new branch] gh/ZolotukhinM/472/base -> origin/gh/ZolotukhinM/472/base 2022-03-17T19:07:02.9118266Z * [new branch] gh/ZolotukhinM/472/head -> origin/gh/ZolotukhinM/472/head 2022-03-17T19:07:02.9120206Z * [new branch] gh/ZolotukhinM/472/orig -> origin/gh/ZolotukhinM/472/orig 2022-03-17T19:07:02.9121344Z * [new branch] gh/ZolotukhinM/473/base -> origin/gh/ZolotukhinM/473/base 2022-03-17T19:07:02.9122591Z * [new branch] gh/ZolotukhinM/473/head -> origin/gh/ZolotukhinM/473/head 2022-03-17T19:07:02.9123809Z * [new branch] gh/ZolotukhinM/473/orig -> origin/gh/ZolotukhinM/473/orig 2022-03-17T19:07:02.9125402Z * [new branch] gh/ZolotukhinM/474/base -> origin/gh/ZolotukhinM/474/base 2022-03-17T19:07:02.9126683Z * [new branch] gh/ZolotukhinM/474/head -> origin/gh/ZolotukhinM/474/head 2022-03-17T19:07:02.9128268Z * [new branch] gh/ZolotukhinM/474/orig -> origin/gh/ZolotukhinM/474/orig 2022-03-17T19:07:02.9130467Z * [new branch] gh/ZolotukhinM/475/base -> origin/gh/ZolotukhinM/475/base 2022-03-17T19:07:02.9131652Z * [new branch] gh/ZolotukhinM/475/head -> origin/gh/ZolotukhinM/475/head 2022-03-17T19:07:02.9133057Z * [new branch] gh/ZolotukhinM/475/orig -> origin/gh/ZolotukhinM/475/orig 2022-03-17T19:07:02.9135016Z * [new branch] gh/ZolotukhinM/482/base -> origin/gh/ZolotukhinM/482/base 2022-03-17T19:07:02.9136138Z * [new branch] gh/ZolotukhinM/482/head -> origin/gh/ZolotukhinM/482/head 2022-03-17T19:07:02.9137482Z * [new branch] gh/ZolotukhinM/482/orig -> origin/gh/ZolotukhinM/482/orig 2022-03-17T19:07:02.9139739Z * [new branch] gh/ZolotukhinM/483/base -> origin/gh/ZolotukhinM/483/base 2022-03-17T19:07:02.9140975Z * [new branch] gh/ZolotukhinM/483/head -> origin/gh/ZolotukhinM/483/head 2022-03-17T19:07:02.9143340Z * [new branch] gh/ZolotukhinM/483/orig -> origin/gh/ZolotukhinM/483/orig 2022-03-17T19:07:02.9145136Z * [new branch] gh/ZolotukhinM/484/base -> origin/gh/ZolotukhinM/484/base 2022-03-17T19:07:02.9146383Z * [new branch] gh/ZolotukhinM/484/head -> origin/gh/ZolotukhinM/484/head 2022-03-17T19:07:02.9147713Z * [new branch] gh/ZolotukhinM/484/orig -> origin/gh/ZolotukhinM/484/orig 2022-03-17T19:07:02.9149937Z * [new branch] gh/ZolotukhinM/485/base -> origin/gh/ZolotukhinM/485/base 2022-03-17T19:07:02.9150867Z * [new branch] gh/ZolotukhinM/485/head -> origin/gh/ZolotukhinM/485/head 2022-03-17T19:07:02.9152446Z * [new branch] gh/ZolotukhinM/485/orig -> origin/gh/ZolotukhinM/485/orig 2022-03-17T19:07:02.9154413Z * [new branch] gh/ZolotukhinM/489/base -> origin/gh/ZolotukhinM/489/base 2022-03-17T19:07:02.9155509Z * [new branch] gh/ZolotukhinM/489/head -> origin/gh/ZolotukhinM/489/head 2022-03-17T19:07:02.9156818Z * [new branch] gh/ZolotukhinM/489/orig -> origin/gh/ZolotukhinM/489/orig 2022-03-17T19:07:02.9159026Z * [new branch] gh/ZolotukhinM/503/base -> origin/gh/ZolotukhinM/503/base 2022-03-17T19:07:02.9160195Z * [new branch] gh/ZolotukhinM/503/head -> origin/gh/ZolotukhinM/503/head 2022-03-17T19:07:02.9161600Z * [new branch] gh/ZolotukhinM/503/orig -> origin/gh/ZolotukhinM/503/orig 2022-03-17T19:07:02.9163761Z * [new branch] gh/ZolotukhinM/506/base -> origin/gh/ZolotukhinM/506/base 2022-03-17T19:07:02.9164897Z * [new branch] gh/ZolotukhinM/506/head -> origin/gh/ZolotukhinM/506/head 2022-03-17T19:07:02.9166242Z * [new branch] gh/ZolotukhinM/506/orig -> origin/gh/ZolotukhinM/506/orig 2022-03-17T19:07:02.9168181Z * [new branch] gh/ZolotukhinM/66/base -> origin/gh/ZolotukhinM/66/base 2022-03-17T19:07:02.9170140Z * [new branch] gh/aazzolini/1/base -> origin/gh/aazzolini/1/base 2022-03-17T19:07:02.9172108Z * [new branch] gh/aazzolini/3/base -> origin/gh/aazzolini/3/base 2022-03-17T19:07:02.9173280Z * [new branch] gh/aazzolini/3/head -> origin/gh/aazzolini/3/head 2022-03-17T19:07:02.9174485Z * [new branch] gh/aazzolini/3/orig -> origin/gh/aazzolini/3/orig 2022-03-17T19:07:02.9176883Z * [new branch] gh/agolynski/1/base -> origin/gh/agolynski/1/base 2022-03-17T19:07:02.9178022Z * [new branch] gh/agolynski/1/head -> origin/gh/agolynski/1/head 2022-03-17T19:07:02.9179319Z * [new branch] gh/agolynski/1/orig -> origin/gh/agolynski/1/orig 2022-03-17T19:07:02.9181360Z * [new branch] gh/agolynski/10/base -> origin/gh/agolynski/10/base 2022-03-17T19:07:02.9182696Z * [new branch] gh/agolynski/10/head -> origin/gh/agolynski/10/head 2022-03-17T19:07:02.9183974Z * [new branch] gh/agolynski/10/orig -> origin/gh/agolynski/10/orig 2022-03-17T19:07:02.9186031Z * [new branch] gh/agolynski/11/base -> origin/gh/agolynski/11/base 2022-03-17T19:07:02.9187209Z * [new branch] gh/agolynski/11/head -> origin/gh/agolynski/11/head 2022-03-17T19:07:02.9188567Z * [new branch] gh/agolynski/11/orig -> origin/gh/agolynski/11/orig 2022-03-17T19:07:02.9189953Z * [new branch] gh/agolynski/2/base -> origin/gh/agolynski/2/base 2022-03-17T19:07:02.9191274Z * [new branch] gh/agolynski/2/head -> origin/gh/agolynski/2/head 2022-03-17T19:07:02.9192736Z * [new branch] gh/agolynski/2/orig -> origin/gh/agolynski/2/orig 2022-03-17T19:07:02.9194870Z * [new branch] gh/agolynski/21/base -> origin/gh/agolynski/21/base 2022-03-17T19:07:02.9196023Z * [new branch] gh/agolynski/21/head -> origin/gh/agolynski/21/head 2022-03-17T19:07:02.9197272Z * [new branch] gh/agolynski/21/orig -> origin/gh/agolynski/21/orig 2022-03-17T19:07:02.9199935Z * [new branch] gh/agolynski/22/base -> origin/gh/agolynski/22/base 2022-03-17T19:07:02.9201118Z * [new branch] gh/agolynski/22/head -> origin/gh/agolynski/22/head 2022-03-17T19:07:02.9202420Z * [new branch] gh/agolynski/22/orig -> origin/gh/agolynski/22/orig 2022-03-17T19:07:02.9204427Z * [new branch] gh/agolynski/6/base -> origin/gh/agolynski/6/base 2022-03-17T19:07:02.9205544Z * [new branch] gh/agolynski/6/head -> origin/gh/agolynski/6/head 2022-03-17T19:07:02.9206931Z * [new branch] gh/agolynski/6/orig -> origin/gh/agolynski/6/orig 2022-03-17T19:07:02.9208849Z * [new branch] gh/agolynski/8/base -> origin/gh/agolynski/8/base 2022-03-17T19:07:02.9210013Z * [new branch] gh/agolynski/8/orig -> origin/gh/agolynski/8/orig 2022-03-17T19:07:02.9212237Z * [new branch] gh/ailzhang/10/base -> origin/gh/ailzhang/10/base 2022-03-17T19:07:02.9213592Z * [new branch] gh/ailzhang/10/head -> origin/gh/ailzhang/10/head 2022-03-17T19:07:02.9214889Z * [new branch] gh/ailzhang/10/orig -> origin/gh/ailzhang/10/orig 2022-03-17T19:07:02.9216962Z * [new branch] gh/ailzhang/11/base -> origin/gh/ailzhang/11/base 2022-03-17T19:07:02.9218127Z * [new branch] gh/ailzhang/11/head -> origin/gh/ailzhang/11/head 2022-03-17T19:07:02.9219914Z * [new branch] gh/ailzhang/11/orig -> origin/gh/ailzhang/11/orig 2022-03-17T19:07:02.9221486Z * [new branch] gh/ailzhang/12/base -> origin/gh/ailzhang/12/base 2022-03-17T19:07:02.9222845Z * [new branch] gh/ailzhang/12/head -> origin/gh/ailzhang/12/head 2022-03-17T19:07:02.9224274Z * [new branch] gh/ailzhang/12/orig -> origin/gh/ailzhang/12/orig 2022-03-17T19:07:02.9227001Z * [new branch] gh/ailzhang/25/base -> origin/gh/ailzhang/25/base 2022-03-17T19:07:02.9227821Z * [new branch] gh/ailzhang/25/head -> origin/gh/ailzhang/25/head 2022-03-17T19:07:02.9229397Z * [new branch] gh/ailzhang/25/orig -> origin/gh/ailzhang/25/orig 2022-03-17T19:07:02.9231318Z * [new branch] gh/ailzhang/50/base -> origin/gh/ailzhang/50/base 2022-03-17T19:07:02.9232459Z * [new branch] gh/ailzhang/50/head -> origin/gh/ailzhang/50/head 2022-03-17T19:07:02.9234273Z * [new branch] gh/ailzhang/50/orig -> origin/gh/ailzhang/50/orig 2022-03-17T19:07:02.9235625Z * [new branch] gh/ailzhang/9/base -> origin/gh/ailzhang/9/base 2022-03-17T19:07:02.9236794Z * [new branch] gh/ailzhang/9/head -> origin/gh/ailzhang/9/head 2022-03-17T19:07:02.9238139Z * [new branch] gh/ailzhang/9/orig -> origin/gh/ailzhang/9/orig 2022-03-17T19:07:02.9240450Z * [new branch] gh/ajtulloch/1/base -> origin/gh/ajtulloch/1/base 2022-03-17T19:07:02.9241602Z * [new branch] gh/ajtulloch/1/head -> origin/gh/ajtulloch/1/head 2022-03-17T19:07:02.9243196Z * [new branch] gh/ajtulloch/2/base -> origin/gh/ajtulloch/2/base 2022-03-17T19:07:02.9244536Z * [new branch] gh/ajtulloch/2/head -> origin/gh/ajtulloch/2/head 2022-03-17T19:07:02.9246687Z * [new branch] gh/alanadakotashine/1/base -> origin/gh/alanadakotashine/1/base 2022-03-17T19:07:02.9247815Z * [new branch] gh/alanadakotashine/1/head -> origin/gh/alanadakotashine/1/head 2022-03-17T19:07:02.9249102Z * [new branch] gh/alanadakotashine/1/orig -> origin/gh/alanadakotashine/1/orig 2022-03-17T19:07:02.9251289Z * [new branch] gh/alanadakotashine/2/base -> origin/gh/alanadakotashine/2/base 2022-03-17T19:07:02.9252334Z * [new branch] gh/alanadakotashine/2/head -> origin/gh/alanadakotashine/2/head 2022-03-17T19:07:02.9253743Z * [new branch] gh/alanadakotashine/2/orig -> origin/gh/alanadakotashine/2/orig 2022-03-17T19:07:02.9256268Z * [new branch] gh/alanwaketan/10/base -> origin/gh/alanwaketan/10/base 2022-03-17T19:07:02.9257416Z * [new branch] gh/alanwaketan/10/head -> origin/gh/alanwaketan/10/head 2022-03-17T19:07:02.9258779Z * [new branch] gh/alanwaketan/10/orig -> origin/gh/alanwaketan/10/orig 2022-03-17T19:07:02.9260925Z * [new branch] gh/alanwaketan/15/base -> origin/gh/alanwaketan/15/base 2022-03-17T19:07:02.9262224Z * [new branch] gh/alanwaketan/15/orig -> origin/gh/alanwaketan/15/orig 2022-03-17T19:07:02.9264334Z * [new branch] gh/alanwaketan/4/base -> origin/gh/alanwaketan/4/base 2022-03-17T19:07:02.9265549Z * [new branch] gh/alanwaketan/4/head -> origin/gh/alanwaketan/4/head 2022-03-17T19:07:02.9266929Z * [new branch] gh/alanwaketan/4/orig -> origin/gh/alanwaketan/4/orig 2022-03-17T19:07:02.9269352Z * [new branch] gh/albanD/4/base -> origin/gh/albanD/4/base 2022-03-17T19:07:02.9270533Z * [new branch] gh/albanD/4/orig -> origin/gh/albanD/4/orig 2022-03-17T19:07:02.9273126Z * [new branch] gh/albanD/92/base -> origin/gh/albanD/92/base 2022-03-17T19:07:02.9274252Z * [new branch] gh/albanD/92/head -> origin/gh/albanD/92/head 2022-03-17T19:07:02.9275638Z * [new branch] gh/albanD/92/orig -> origin/gh/albanD/92/orig 2022-03-17T19:07:02.9277596Z * [new branch] gh/albanD/94/base -> origin/gh/albanD/94/base 2022-03-17T19:07:02.9278792Z * [new branch] gh/albanD/94/head -> origin/gh/albanD/94/head 2022-03-17T19:07:02.9280155Z * [new branch] gh/albanD/94/orig -> origin/gh/albanD/94/orig 2022-03-17T19:07:02.9282222Z * [new branch] gh/albanD/96/base -> origin/gh/albanD/96/base 2022-03-17T19:07:02.9283255Z * [new branch] gh/albanD/96/head -> origin/gh/albanD/96/head 2022-03-17T19:07:02.9284579Z * [new branch] gh/albanD/96/orig -> origin/gh/albanD/96/orig 2022-03-17T19:07:02.9287178Z * [new branch] gh/andrewor14/1/base -> origin/gh/andrewor14/1/base 2022-03-17T19:07:02.9288371Z * [new branch] gh/andrewor14/1/orig -> origin/gh/andrewor14/1/orig 2022-03-17T19:07:02.9290342Z * [new branch] gh/andrewor14/10/base -> origin/gh/andrewor14/10/base 2022-03-17T19:07:02.9291604Z * [new branch] gh/andrewor14/10/orig -> origin/gh/andrewor14/10/orig 2022-03-17T19:07:02.9293619Z * [new branch] gh/andrewor14/11/base -> origin/gh/andrewor14/11/base 2022-03-17T19:07:02.9294800Z * [new branch] gh/andrewor14/11/orig -> origin/gh/andrewor14/11/orig 2022-03-17T19:07:02.9296801Z * [new branch] gh/andrewor14/12/base -> origin/gh/andrewor14/12/base 2022-03-17T19:07:02.9298013Z * [new branch] gh/andrewor14/12/head -> origin/gh/andrewor14/12/head 2022-03-17T19:07:02.9299255Z * [new branch] gh/andrewor14/12/orig -> origin/gh/andrewor14/12/orig 2022-03-17T19:07:02.9301146Z * [new branch] gh/andrewor14/2/base -> origin/gh/andrewor14/2/base 2022-03-17T19:07:02.9302778Z * [new branch] gh/andrewor14/2/head -> origin/gh/andrewor14/2/head 2022-03-17T19:07:02.9304089Z * [new branch] gh/andrewor14/2/orig -> origin/gh/andrewor14/2/orig 2022-03-17T19:07:02.9306345Z * [new branch] gh/andrewor14/3/base -> origin/gh/andrewor14/3/base 2022-03-17T19:07:02.9307589Z * [new branch] gh/andrewor14/3/orig -> origin/gh/andrewor14/3/orig 2022-03-17T19:07:02.9309561Z * [new branch] gh/andrewor14/4/base -> origin/gh/andrewor14/4/base 2022-03-17T19:07:02.9310479Z * [new branch] gh/andrewor14/4/orig -> origin/gh/andrewor14/4/orig 2022-03-17T19:07:02.9312630Z * [new branch] gh/andrewor14/5/base -> origin/gh/andrewor14/5/base 2022-03-17T19:07:02.9313802Z * [new branch] gh/andrewor14/5/orig -> origin/gh/andrewor14/5/orig 2022-03-17T19:07:02.9315738Z * [new branch] gh/andrewor14/6/base -> origin/gh/andrewor14/6/base 2022-03-17T19:07:02.9317703Z * [new branch] gh/andrewor14/6/orig -> origin/gh/andrewor14/6/orig 2022-03-17T19:07:02.9319523Z * [new branch] gh/andrewor14/7/base -> origin/gh/andrewor14/7/base 2022-03-17T19:07:02.9320701Z * [new branch] gh/andrewor14/7/orig -> origin/gh/andrewor14/7/orig 2022-03-17T19:07:02.9322622Z * [new branch] gh/andrewor14/8/base -> origin/gh/andrewor14/8/base 2022-03-17T19:07:02.9323789Z * [new branch] gh/andrewor14/8/orig -> origin/gh/andrewor14/8/orig 2022-03-17T19:07:02.9326032Z * [new branch] gh/andrewor14/9/base -> origin/gh/andrewor14/9/base 2022-03-17T19:07:02.9327167Z * [new branch] gh/andrewor14/9/orig -> origin/gh/andrewor14/9/orig 2022-03-17T19:07:02.9329403Z * [new branch] gh/angelayi/18/base -> origin/gh/angelayi/18/base 2022-03-17T19:07:02.9330594Z * [new branch] gh/angelayi/18/head -> origin/gh/angelayi/18/head 2022-03-17T19:07:02.9332199Z * [new branch] gh/angelayi/18/orig -> origin/gh/angelayi/18/orig 2022-03-17T19:07:02.9334786Z * [new branch] gh/angelayi/33/base -> origin/gh/angelayi/33/base 2022-03-17T19:07:02.9336050Z * [new branch] gh/angelayi/33/head -> origin/gh/angelayi/33/head 2022-03-17T19:07:02.9337736Z * [new branch] gh/angelayi/33/orig -> origin/gh/angelayi/33/orig 2022-03-17T19:07:02.9339614Z * [new branch] gh/angelayi/34/base -> origin/gh/angelayi/34/base 2022-03-17T19:07:02.9340979Z * [new branch] gh/angelayi/34/head -> origin/gh/angelayi/34/head 2022-03-17T19:07:02.9342841Z * [new branch] gh/angelayi/34/orig -> origin/gh/angelayi/34/orig 2022-03-17T19:07:02.9345019Z * [new branch] gh/anijain2305/6/base -> origin/gh/anijain2305/6/base 2022-03-17T19:07:02.9346162Z * [new branch] gh/anijain2305/6/head -> origin/gh/anijain2305/6/head 2022-03-17T19:07:02.9347569Z * [new branch] gh/anijain2305/6/orig -> origin/gh/anijain2305/6/orig 2022-03-17T19:07:02.9349625Z * [new branch] gh/anjali411/100/base -> origin/gh/anjali411/100/base 2022-03-17T19:07:02.9350834Z * [new branch] gh/anjali411/100/head -> origin/gh/anjali411/100/head 2022-03-17T19:07:02.9352430Z * [new branch] gh/anjali411/100/orig -> origin/gh/anjali411/100/orig 2022-03-17T19:07:02.9354176Z * [new branch] gh/anjali411/106/base -> origin/gh/anjali411/106/base 2022-03-17T19:07:02.9355344Z * [new branch] gh/anjali411/106/head -> origin/gh/anjali411/106/head 2022-03-17T19:07:02.9356901Z * [new branch] gh/anjali411/106/orig -> origin/gh/anjali411/106/orig 2022-03-17T19:07:02.9358760Z * [new branch] gh/anjali411/124/base -> origin/gh/anjali411/124/base 2022-03-17T19:07:02.9359834Z * [new branch] gh/anjali411/124/head -> origin/gh/anjali411/124/head 2022-03-17T19:07:02.9361254Z * [new branch] gh/anjali411/124/orig -> origin/gh/anjali411/124/orig 2022-03-17T19:07:02.9363423Z * [new branch] gh/anjali411/128/base -> origin/gh/anjali411/128/base 2022-03-17T19:07:02.9364619Z * [new branch] gh/anjali411/128/head -> origin/gh/anjali411/128/head 2022-03-17T19:07:02.9366001Z * [new branch] gh/anjali411/128/orig -> origin/gh/anjali411/128/orig 2022-03-17T19:07:02.9367986Z * [new branch] gh/anjali411/13/base -> origin/gh/anjali411/13/base 2022-03-17T19:07:02.9369183Z * [new branch] gh/anjali411/13/head -> origin/gh/anjali411/13/head 2022-03-17T19:07:02.9370831Z * [new branch] gh/anjali411/13/orig -> origin/gh/anjali411/13/orig 2022-03-17T19:07:02.9372545Z * [new branch] gh/anjali411/130/base -> origin/gh/anjali411/130/base 2022-03-17T19:07:02.9373779Z * [new branch] gh/anjali411/130/head -> origin/gh/anjali411/130/head 2022-03-17T19:07:02.9375382Z * [new branch] gh/anjali411/130/orig -> origin/gh/anjali411/130/orig 2022-03-17T19:07:02.9377221Z * [new branch] gh/anjali411/14/base -> origin/gh/anjali411/14/base 2022-03-17T19:07:02.9378245Z * [new branch] gh/anjali411/14/orig -> origin/gh/anjali411/14/orig 2022-03-17T19:07:02.9380307Z * [new branch] gh/anjali411/145/base -> origin/gh/anjali411/145/base 2022-03-17T19:07:02.9383972Z * [new branch] gh/anjali411/145/head -> origin/gh/anjali411/145/head 2022-03-17T19:07:02.9384964Z * [new branch] gh/anjali411/145/orig -> origin/gh/anjali411/145/orig 2022-03-17T19:07:02.9387073Z * [new branch] gh/anjali411/153/base -> origin/gh/anjali411/153/base 2022-03-17T19:07:02.9388247Z * [new branch] gh/anjali411/153/head -> origin/gh/anjali411/153/head 2022-03-17T19:07:02.9389841Z * [new branch] gh/anjali411/153/orig -> origin/gh/anjali411/153/orig 2022-03-17T19:07:02.9391549Z * [new branch] gh/anjali411/155/base -> origin/gh/anjali411/155/base 2022-03-17T19:07:02.9393583Z * [new branch] gh/anjali411/155/head -> origin/gh/anjali411/155/head 2022-03-17T19:07:02.9394798Z * [new branch] gh/anjali411/155/orig -> origin/gh/anjali411/155/orig 2022-03-17T19:07:02.9396766Z * [new branch] gh/anjali411/156/base -> origin/gh/anjali411/156/base 2022-03-17T19:07:02.9397809Z * [new branch] gh/anjali411/156/head -> origin/gh/anjali411/156/head 2022-03-17T19:07:02.9399424Z * [new branch] gh/anjali411/156/orig -> origin/gh/anjali411/156/orig 2022-03-17T19:07:02.9401120Z * [new branch] gh/anjali411/159/base -> origin/gh/anjali411/159/base 2022-03-17T19:07:02.9402287Z * [new branch] gh/anjali411/159/head -> origin/gh/anjali411/159/head 2022-03-17T19:07:02.9404047Z * [new branch] gh/anjali411/159/orig -> origin/gh/anjali411/159/orig 2022-03-17T19:07:02.9405791Z * [new branch] gh/anjali411/160/base -> origin/gh/anjali411/160/base 2022-03-17T19:07:02.9406922Z * [new branch] gh/anjali411/160/head -> origin/gh/anjali411/160/head 2022-03-17T19:07:02.9408560Z * [new branch] gh/anjali411/160/orig -> origin/gh/anjali411/160/orig 2022-03-17T19:07:02.9410285Z * [new branch] gh/anjali411/19/base -> origin/gh/anjali411/19/base 2022-03-17T19:07:02.9411422Z * [new branch] gh/anjali411/19/head -> origin/gh/anjali411/19/head 2022-03-17T19:07:02.9412801Z * [new branch] gh/anjali411/19/orig -> origin/gh/anjali411/19/orig 2022-03-17T19:07:02.9414735Z * [new branch] gh/anjali411/22/base -> origin/gh/anjali411/22/base 2022-03-17T19:07:02.9416914Z * [new branch] gh/anjali411/22/head -> origin/gh/anjali411/22/head 2022-03-17T19:07:02.9417348Z * [new branch] gh/anjali411/22/orig -> origin/gh/anjali411/22/orig 2022-03-17T19:07:02.9419313Z * [new branch] gh/anjali411/25/base -> origin/gh/anjali411/25/base 2022-03-17T19:07:02.9420670Z * [new branch] gh/anjali411/25/head -> origin/gh/anjali411/25/head 2022-03-17T19:07:02.9422404Z * [new branch] gh/anjali411/25/orig -> origin/gh/anjali411/25/orig 2022-03-17T19:07:02.9424133Z * [new branch] gh/anjali411/26/base -> origin/gh/anjali411/26/base 2022-03-17T19:07:02.9425307Z * [new branch] gh/anjali411/26/head -> origin/gh/anjali411/26/head 2022-03-17T19:07:02.9426907Z * [new branch] gh/anjali411/26/orig -> origin/gh/anjali411/26/orig 2022-03-17T19:07:02.9429210Z * [new branch] gh/anjali411/31/base -> origin/gh/anjali411/31/base 2022-03-17T19:07:02.9430397Z * [new branch] gh/anjali411/31/head -> origin/gh/anjali411/31/head 2022-03-17T19:07:02.9431810Z * [new branch] gh/anjali411/31/orig -> origin/gh/anjali411/31/orig 2022-03-17T19:07:02.9433769Z * [new branch] gh/anjali411/68/base -> origin/gh/anjali411/68/base 2022-03-17T19:07:02.9434815Z * [new branch] gh/anjali411/68/head -> origin/gh/anjali411/68/head 2022-03-17T19:07:02.9436478Z * [new branch] gh/anjali411/68/orig -> origin/gh/anjali411/68/orig 2022-03-17T19:07:02.9438743Z * [new branch] gh/anjali411/77/base -> origin/gh/anjali411/77/base 2022-03-17T19:07:02.9440753Z * [new branch] gh/anjali411/77/head -> origin/gh/anjali411/77/head 2022-03-17T19:07:02.9441931Z * [new branch] gh/anjali411/77/orig -> origin/gh/anjali411/77/orig 2022-03-17T19:07:02.9444105Z * [new branch] gh/anjali411/80/base -> origin/gh/anjali411/80/base 2022-03-17T19:07:02.9445250Z * [new branch] gh/anjali411/80/orig -> origin/gh/anjali411/80/orig 2022-03-17T19:07:02.9447446Z * [new branch] gh/ansley/14/base -> origin/gh/ansley/14/base 2022-03-17T19:07:02.9448478Z * [new branch] gh/ansley/14/head -> origin/gh/ansley/14/head 2022-03-17T19:07:02.9450200Z * [new branch] gh/ansley/14/orig -> origin/gh/ansley/14/orig 2022-03-17T19:07:02.9451842Z * [new branch] gh/ansley/24/base -> origin/gh/ansley/24/base 2022-03-17T19:07:02.9452963Z * [new branch] gh/ansley/24/head -> origin/gh/ansley/24/head 2022-03-17T19:07:02.9454754Z * [new branch] gh/ansley/24/orig -> origin/gh/ansley/24/orig 2022-03-17T19:07:02.9456664Z * [new branch] gh/ansley/25/base -> origin/gh/ansley/25/base 2022-03-17T19:07:02.9457768Z * [new branch] gh/ansley/25/head -> origin/gh/ansley/25/head 2022-03-17T19:07:02.9459056Z * [new branch] gh/ansley/25/orig -> origin/gh/ansley/25/orig 2022-03-17T19:07:02.9461044Z * [new branch] gh/ansley/26/base -> origin/gh/ansley/26/base 2022-03-17T19:07:02.9462899Z * [new branch] gh/ansley/26/head -> origin/gh/ansley/26/head 2022-03-17T19:07:02.9463974Z * [new branch] gh/ansley/26/orig -> origin/gh/ansley/26/orig 2022-03-17T19:07:02.9465855Z * [new branch] gh/ansley/27/base -> origin/gh/ansley/27/base 2022-03-17T19:07:02.9466880Z * [new branch] gh/ansley/27/head -> origin/gh/ansley/27/head 2022-03-17T19:07:02.9468250Z * [new branch] gh/ansley/27/orig -> origin/gh/ansley/27/orig 2022-03-17T19:07:02.9470343Z * [new branch] gh/ansley/28/base -> origin/gh/ansley/28/base 2022-03-17T19:07:02.9471498Z * [new branch] gh/ansley/28/head -> origin/gh/ansley/28/head 2022-03-17T19:07:02.9472908Z * [new branch] gh/ansley/28/orig -> origin/gh/ansley/28/orig 2022-03-17T19:07:02.9474896Z * [new branch] gh/ansley/30/base -> origin/gh/ansley/30/base 2022-03-17T19:07:02.9475967Z * [new branch] gh/ansley/30/head -> origin/gh/ansley/30/head 2022-03-17T19:07:02.9477321Z * [new branch] gh/ansley/30/orig -> origin/gh/ansley/30/orig 2022-03-17T19:07:02.9479334Z * [new branch] gh/ansley/35/base -> origin/gh/ansley/35/base 2022-03-17T19:07:02.9480359Z * [new branch] gh/ansley/35/head -> origin/gh/ansley/35/head 2022-03-17T19:07:02.9482239Z * [new branch] gh/ansley/35/orig -> origin/gh/ansley/35/orig 2022-03-17T19:07:02.9483618Z * [new branch] gh/ansley/36/base -> origin/gh/ansley/36/base 2022-03-17T19:07:02.9485185Z * [new branch] gh/ansley/36/head -> origin/gh/ansley/36/head 2022-03-17T19:07:02.9486154Z * [new branch] gh/ansley/36/orig -> origin/gh/ansley/36/orig 2022-03-17T19:07:02.9489179Z * [new branch] gh/ansley/38/base -> origin/gh/ansley/38/base 2022-03-17T19:07:02.9490154Z * [new branch] gh/ansley/38/head -> origin/gh/ansley/38/head 2022-03-17T19:07:02.9491846Z * [new branch] gh/ansley/38/orig -> origin/gh/ansley/38/orig 2022-03-17T19:07:02.9493598Z * [new branch] gh/ansley/41/base -> origin/gh/ansley/41/base 2022-03-17T19:07:02.9494642Z * [new branch] gh/ansley/41/head -> origin/gh/ansley/41/head 2022-03-17T19:07:02.9496272Z * [new branch] gh/ansley/41/orig -> origin/gh/ansley/41/orig 2022-03-17T19:07:02.9498168Z * [new branch] gh/ansley/43/base -> origin/gh/ansley/43/base 2022-03-17T19:07:02.9499215Z * [new branch] gh/ansley/43/head -> origin/gh/ansley/43/head 2022-03-17T19:07:02.9500782Z * [new branch] gh/ansley/43/orig -> origin/gh/ansley/43/orig 2022-03-17T19:07:02.9502992Z * [new branch] gh/ansley/45/base -> origin/gh/ansley/45/base 2022-03-17T19:07:02.9504050Z * [new branch] gh/ansley/45/head -> origin/gh/ansley/45/head 2022-03-17T19:07:02.9505380Z * [new branch] gh/ansley/45/orig -> origin/gh/ansley/45/orig 2022-03-17T19:07:02.9507461Z * [new branch] gh/ansley/47/base -> origin/gh/ansley/47/base 2022-03-17T19:07:02.9509527Z * [new branch] gh/ansley/47/head -> origin/gh/ansley/47/head 2022-03-17T19:07:02.9510595Z * [new branch] gh/ansley/47/orig -> origin/gh/ansley/47/orig 2022-03-17T19:07:02.9512662Z * [new branch] gh/ansley/48/base -> origin/gh/ansley/48/base 2022-03-17T19:07:02.9513662Z * [new branch] gh/ansley/48/head -> origin/gh/ansley/48/head 2022-03-17T19:07:02.9514976Z * [new branch] gh/ansley/48/orig -> origin/gh/ansley/48/orig 2022-03-17T19:07:02.9516942Z * [new branch] gh/ansley/51/base -> origin/gh/ansley/51/base 2022-03-17T19:07:02.9518037Z * [new branch] gh/ansley/51/head -> origin/gh/ansley/51/head 2022-03-17T19:07:02.9519430Z * [new branch] gh/ansley/51/orig -> origin/gh/ansley/51/orig 2022-03-17T19:07:02.9521719Z * [new branch] gh/ansley/54/base -> origin/gh/ansley/54/base 2022-03-17T19:07:02.9522711Z * [new branch] gh/ansley/54/head -> origin/gh/ansley/54/head 2022-03-17T19:07:02.9524264Z * [new branch] gh/ansley/54/orig -> origin/gh/ansley/54/orig 2022-03-17T19:07:02.9525904Z * [new branch] gh/ansley/55/base -> origin/gh/ansley/55/base 2022-03-17T19:07:02.9527207Z * [new branch] gh/ansley/55/head -> origin/gh/ansley/55/head 2022-03-17T19:07:02.9528830Z * [new branch] gh/ansley/55/orig -> origin/gh/ansley/55/orig 2022-03-17T19:07:02.9530602Z * [new branch] gh/ansley/56/base -> origin/gh/ansley/56/base 2022-03-17T19:07:02.9531603Z * [new branch] gh/ansley/56/head -> origin/gh/ansley/56/head 2022-03-17T19:07:02.9532988Z * [new branch] gh/ansley/56/orig -> origin/gh/ansley/56/orig 2022-03-17T19:07:02.9535468Z * [new branch] gh/ansley/57/base -> origin/gh/ansley/57/base 2022-03-17T19:07:02.9536671Z * [new branch] gh/ansley/57/head -> origin/gh/ansley/57/head 2022-03-17T19:07:02.9538048Z * [new branch] gh/ansley/57/orig -> origin/gh/ansley/57/orig 2022-03-17T19:07:02.9539655Z * [new branch] gh/ansley/58/base -> origin/gh/ansley/58/base 2022-03-17T19:07:02.9541065Z * [new branch] gh/ansley/58/head -> origin/gh/ansley/58/head 2022-03-17T19:07:02.9542992Z * [new branch] gh/ansley/58/orig -> origin/gh/ansley/58/orig 2022-03-17T19:07:02.9544208Z * [new branch] gh/ansley/59/base -> origin/gh/ansley/59/base 2022-03-17T19:07:02.9545492Z * [new branch] gh/ansley/59/head -> origin/gh/ansley/59/head 2022-03-17T19:07:02.9546794Z * [new branch] gh/ansley/59/orig -> origin/gh/ansley/59/orig 2022-03-17T19:07:02.9549162Z * [new branch] gh/ansleyadelaide/1/base -> origin/gh/ansleyadelaide/1/base 2022-03-17T19:07:02.9550284Z * [new branch] gh/ansleyadelaide/1/orig -> origin/gh/ansleyadelaide/1/orig 2022-03-17T19:07:02.9552311Z * [new branch] gh/ansleyadelaide/10/base -> origin/gh/ansleyadelaide/10/base 2022-03-17T19:07:02.9553410Z * [new branch] gh/ansleyadelaide/10/orig -> origin/gh/ansleyadelaide/10/orig 2022-03-17T19:07:02.9555396Z * [new branch] gh/ansleyadelaide/11/base -> origin/gh/ansleyadelaide/11/base 2022-03-17T19:07:02.9556418Z * [new branch] gh/ansleyadelaide/11/head -> origin/gh/ansleyadelaide/11/head 2022-03-17T19:07:02.9558777Z * [new branch] gh/ansleyadelaide/14/base -> origin/gh/ansleyadelaide/14/base 2022-03-17T19:07:02.9559839Z * [new branch] gh/ansleyadelaide/14/orig -> origin/gh/ansleyadelaide/14/orig 2022-03-17T19:07:02.9561963Z * [new branch] gh/ansleyadelaide/18/base -> origin/gh/ansleyadelaide/18/base 2022-03-17T19:07:02.9563036Z * [new branch] gh/ansleyadelaide/18/head -> origin/gh/ansleyadelaide/18/head 2022-03-17T19:07:02.9564364Z * [new branch] gh/ansleyadelaide/18/orig -> origin/gh/ansleyadelaide/18/orig 2022-03-17T19:07:02.9566261Z * [new branch] gh/ansleyadelaide/2/base -> origin/gh/ansleyadelaide/2/base 2022-03-17T19:07:02.9567286Z * [new branch] gh/ansleyadelaide/2/head -> origin/gh/ansleyadelaide/2/head 2022-03-17T19:07:02.9568667Z * [new branch] gh/ansleyadelaide/2/orig -> origin/gh/ansleyadelaide/2/orig 2022-03-17T19:07:02.9570599Z * [new branch] gh/ansleyadelaide/22/base -> origin/gh/ansleyadelaide/22/base 2022-03-17T19:07:02.9571651Z * [new branch] gh/ansleyadelaide/22/orig -> origin/gh/ansleyadelaide/22/orig 2022-03-17T19:07:02.9573839Z * [new branch] gh/ansleyadelaide/26/base -> origin/gh/ansleyadelaide/26/base 2022-03-17T19:07:02.9574951Z * [new branch] gh/ansleyadelaide/26/orig -> origin/gh/ansleyadelaide/26/orig 2022-03-17T19:07:02.9577238Z * [new branch] gh/ansleyadelaide/28/base -> origin/gh/ansleyadelaide/28/base 2022-03-17T19:07:02.9578327Z * [new branch] gh/ansleyadelaide/28/orig -> origin/gh/ansleyadelaide/28/orig 2022-03-17T19:07:02.9580352Z * [new branch] gh/ansleyadelaide/29/base -> origin/gh/ansleyadelaide/29/base 2022-03-17T19:07:02.9581441Z * [new branch] gh/ansleyadelaide/29/orig -> origin/gh/ansleyadelaide/29/orig 2022-03-17T19:07:02.9584208Z * [new branch] gh/ansleyadelaide/3/base -> origin/gh/ansleyadelaide/3/base 2022-03-17T19:07:02.9585411Z * [new branch] gh/ansleyadelaide/3/head -> origin/gh/ansleyadelaide/3/head 2022-03-17T19:07:02.9586675Z * [new branch] gh/ansleyadelaide/3/orig -> origin/gh/ansleyadelaide/3/orig 2022-03-17T19:07:02.9588716Z * [new branch] gh/ansleyadelaide/31/base -> origin/gh/ansleyadelaide/31/base 2022-03-17T19:07:02.9589697Z * [new branch] gh/ansleyadelaide/31/orig -> origin/gh/ansleyadelaide/31/orig 2022-03-17T19:07:02.9591776Z * [new branch] gh/ansleyadelaide/37/base -> origin/gh/ansleyadelaide/37/base 2022-03-17T19:07:02.9592871Z * [new branch] gh/ansleyadelaide/37/head -> origin/gh/ansleyadelaide/37/head 2022-03-17T19:07:02.9594180Z * [new branch] gh/ansleyadelaide/37/orig -> origin/gh/ansleyadelaide/37/orig 2022-03-17T19:07:02.9596312Z * [new branch] gh/ansleyadelaide/4/base -> origin/gh/ansleyadelaide/4/base 2022-03-17T19:07:02.9597570Z * [new branch] gh/ansleyadelaide/4/head -> origin/gh/ansleyadelaide/4/head 2022-03-17T19:07:02.9598916Z * [new branch] gh/ansleyadelaide/4/orig -> origin/gh/ansleyadelaide/4/orig 2022-03-17T19:07:02.9600873Z * [new branch] gh/ansleyadelaide/43/base -> origin/gh/ansleyadelaide/43/base 2022-03-17T19:07:02.9602304Z * [new branch] gh/ansleyadelaide/9/base -> origin/gh/ansleyadelaide/9/base 2022-03-17T19:07:02.9603679Z * [new branch] gh/ansleyadelaide/9/orig -> origin/gh/ansleyadelaide/9/orig 2022-03-17T19:07:02.9606038Z * [new branch] gh/aocsa/1/base -> origin/gh/aocsa/1/base 2022-03-17T19:07:02.9607084Z * [new branch] gh/aocsa/1/orig -> origin/gh/aocsa/1/orig 2022-03-17T19:07:02.9609112Z * [new branch] gh/aocsa/20/base -> origin/gh/aocsa/20/base 2022-03-17T19:07:02.9610254Z * [new branch] gh/aocsa/20/head -> origin/gh/aocsa/20/head 2022-03-17T19:07:02.9612695Z * [new branch] gh/asalioufb/1/base -> origin/gh/asalioufb/1/base 2022-03-17T19:07:02.9613748Z * [new branch] gh/asalioufb/1/head -> origin/gh/asalioufb/1/head 2022-03-17T19:07:02.9615942Z * [new branch] gh/awgu/14/base -> origin/gh/awgu/14/base 2022-03-17T19:07:02.9616984Z * [new branch] gh/awgu/14/head -> origin/gh/awgu/14/head 2022-03-17T19:07:02.9618352Z * [new branch] gh/awgu/14/orig -> origin/gh/awgu/14/orig 2022-03-17T19:07:02.9619749Z * [new branch] gh/awgu/15/base -> origin/gh/awgu/15/base 2022-03-17T19:07:02.9621177Z * [new branch] gh/awgu/15/head -> origin/gh/awgu/15/head 2022-03-17T19:07:02.9623046Z * [new branch] gh/awgu/15/orig -> origin/gh/awgu/15/orig 2022-03-17T19:07:02.9624212Z * [new branch] gh/awgu/16/base -> origin/gh/awgu/16/base 2022-03-17T19:07:02.9625552Z * [new branch] gh/awgu/16/head -> origin/gh/awgu/16/head 2022-03-17T19:07:02.9626910Z * [new branch] gh/awgu/16/orig -> origin/gh/awgu/16/orig 2022-03-17T19:07:02.9629369Z * [new branch] gh/axitkhurana/10/base -> origin/gh/axitkhurana/10/base 2022-03-17T19:07:02.9630360Z * [new branch] gh/axitkhurana/10/head -> origin/gh/axitkhurana/10/head 2022-03-17T19:07:02.9631710Z * [new branch] gh/axitkhurana/10/orig -> origin/gh/axitkhurana/10/orig 2022-03-17T19:07:02.9633946Z * [new branch] gh/axitkhurana/19/base -> origin/gh/axitkhurana/19/base 2022-03-17T19:07:02.9635087Z * [new branch] gh/axitkhurana/19/head -> origin/gh/axitkhurana/19/head 2022-03-17T19:07:02.9636543Z * [new branch] gh/axitkhurana/19/orig -> origin/gh/axitkhurana/19/orig 2022-03-17T19:07:02.9638637Z * [new branch] gh/axitkhurana/21/base -> origin/gh/axitkhurana/21/base 2022-03-17T19:07:02.9639956Z * [new branch] gh/axitkhurana/21/head -> origin/gh/axitkhurana/21/head 2022-03-17T19:07:02.9641146Z * [new branch] gh/axitkhurana/21/orig -> origin/gh/axitkhurana/21/orig 2022-03-17T19:07:02.9643342Z * [new branch] gh/axitkhurana/23/base -> origin/gh/axitkhurana/23/base 2022-03-17T19:07:02.9644404Z * [new branch] gh/axitkhurana/23/orig -> origin/gh/axitkhurana/23/orig 2022-03-17T19:07:02.9646451Z * [new branch] gh/axitkhurana/6/base -> origin/gh/axitkhurana/6/base 2022-03-17T19:07:02.9647582Z * [new branch] gh/axitkhurana/6/head -> origin/gh/axitkhurana/6/head 2022-03-17T19:07:02.9649509Z * [new branch] gh/axitkhurana/6/orig -> origin/gh/axitkhurana/6/orig 2022-03-17T19:07:02.9651452Z * [new branch] gh/b-koopman/11/base -> origin/gh/b-koopman/11/base 2022-03-17T19:07:02.9652660Z * [new branch] gh/b-koopman/11/head -> origin/gh/b-koopman/11/head 2022-03-17T19:07:02.9654010Z * [new branch] gh/b-koopman/11/orig -> origin/gh/b-koopman/11/orig 2022-03-17T19:07:02.9656141Z * [new branch] gh/b-koopman/13/base -> origin/gh/b-koopman/13/base 2022-03-17T19:07:02.9657293Z * [new branch] gh/b-koopman/13/head -> origin/gh/b-koopman/13/head 2022-03-17T19:07:02.9659177Z * [new branch] gh/b-koopman/13/orig -> origin/gh/b-koopman/13/orig 2022-03-17T19:07:02.9660535Z * [new branch] gh/b-koopman/20/base -> origin/gh/b-koopman/20/base 2022-03-17T19:07:02.9662731Z * [new branch] gh/b-koopman/20/head -> origin/gh/b-koopman/20/head 2022-03-17T19:07:02.9663872Z * [new branch] gh/b-koopman/20/orig -> origin/gh/b-koopman/20/orig 2022-03-17T19:07:02.9666217Z * [new branch] gh/banitag1/1/base -> origin/gh/banitag1/1/base 2022-03-17T19:07:02.9667978Z * [new branch] gh/banitag1/1/head -> origin/gh/banitag1/1/head 2022-03-17T19:07:02.9669319Z * [new branch] gh/banitag1/1/orig -> origin/gh/banitag1/1/orig 2022-03-17T19:07:02.9671698Z * [new branch] gh/bdhirsh/108/base -> origin/gh/bdhirsh/108/base 2022-03-17T19:07:02.9673068Z * [new branch] gh/bdhirsh/108/head -> origin/gh/bdhirsh/108/head 2022-03-17T19:07:02.9674418Z * [new branch] gh/bdhirsh/108/orig -> origin/gh/bdhirsh/108/orig 2022-03-17T19:07:02.9677037Z * [new branch] gh/bdhirsh/109/base -> origin/gh/bdhirsh/109/base 2022-03-17T19:07:02.9678939Z * [new branch] gh/bdhirsh/109/head -> origin/gh/bdhirsh/109/head 2022-03-17T19:07:02.9679985Z * [new branch] gh/bdhirsh/109/orig -> origin/gh/bdhirsh/109/orig 2022-03-17T19:07:02.9681670Z * [new branch] gh/bdhirsh/111/base -> origin/gh/bdhirsh/111/base 2022-03-17T19:07:02.9682800Z * [new branch] gh/bdhirsh/111/head -> origin/gh/bdhirsh/111/head 2022-03-17T19:07:02.9684145Z * [new branch] gh/bdhirsh/111/orig -> origin/gh/bdhirsh/111/orig 2022-03-17T19:07:02.9686076Z * [new branch] gh/bdhirsh/114/base -> origin/gh/bdhirsh/114/base 2022-03-17T19:07:02.9687414Z * [new branch] gh/bdhirsh/114/head -> origin/gh/bdhirsh/114/head 2022-03-17T19:07:02.9688792Z * [new branch] gh/bdhirsh/114/orig -> origin/gh/bdhirsh/114/orig 2022-03-17T19:07:02.9691040Z * [new branch] gh/bdhirsh/115/base -> origin/gh/bdhirsh/115/base 2022-03-17T19:07:02.9692283Z * [new branch] gh/bdhirsh/115/head -> origin/gh/bdhirsh/115/head 2022-03-17T19:07:02.9693736Z * [new branch] gh/bdhirsh/115/orig -> origin/gh/bdhirsh/115/orig 2022-03-17T19:07:02.9695732Z * [new branch] gh/bdhirsh/118/base -> origin/gh/bdhirsh/118/base 2022-03-17T19:07:02.9696874Z * [new branch] gh/bdhirsh/118/head -> origin/gh/bdhirsh/118/head 2022-03-17T19:07:02.9698293Z * [new branch] gh/bdhirsh/118/orig -> origin/gh/bdhirsh/118/orig 2022-03-17T19:07:02.9700267Z * [new branch] gh/bdhirsh/119/base -> origin/gh/bdhirsh/119/base 2022-03-17T19:07:02.9701318Z * [new branch] gh/bdhirsh/119/head -> origin/gh/bdhirsh/119/head 2022-03-17T19:07:02.9703213Z * [new branch] gh/bdhirsh/119/orig -> origin/gh/bdhirsh/119/orig 2022-03-17T19:07:02.9705368Z * [new branch] gh/bdhirsh/128/base -> origin/gh/bdhirsh/128/base 2022-03-17T19:07:02.9706519Z * [new branch] gh/bdhirsh/128/head -> origin/gh/bdhirsh/128/head 2022-03-17T19:07:02.9708427Z * [new branch] gh/bdhirsh/128/orig -> origin/gh/bdhirsh/128/orig 2022-03-17T19:07:02.9709869Z * [new branch] gh/bdhirsh/131/base -> origin/gh/bdhirsh/131/base 2022-03-17T19:07:02.9711628Z * [new branch] gh/bdhirsh/131/head -> origin/gh/bdhirsh/131/head 2022-03-17T19:07:02.9712684Z * [new branch] gh/bdhirsh/131/orig -> origin/gh/bdhirsh/131/orig 2022-03-17T19:07:02.9714854Z * [new branch] gh/bdhirsh/132/base -> origin/gh/bdhirsh/132/base 2022-03-17T19:07:02.9716738Z * [new branch] gh/bdhirsh/132/head -> origin/gh/bdhirsh/132/head 2022-03-17T19:07:02.9717761Z * [new branch] gh/bdhirsh/132/orig -> origin/gh/bdhirsh/132/orig 2022-03-17T19:07:02.9720014Z * [new branch] gh/bdhirsh/133/base -> origin/gh/bdhirsh/133/base 2022-03-17T19:07:02.9721427Z * [new branch] gh/bdhirsh/133/head -> origin/gh/bdhirsh/133/head 2022-03-17T19:07:02.9723190Z * [new branch] gh/bdhirsh/133/orig -> origin/gh/bdhirsh/133/orig 2022-03-17T19:07:02.9725355Z * [new branch] gh/bdhirsh/134/base -> origin/gh/bdhirsh/134/base 2022-03-17T19:07:02.9726504Z * [new branch] gh/bdhirsh/134/head -> origin/gh/bdhirsh/134/head 2022-03-17T19:07:02.9727874Z * [new branch] gh/bdhirsh/134/orig -> origin/gh/bdhirsh/134/orig 2022-03-17T19:07:02.9730216Z * [new branch] gh/bdhirsh/136/base -> origin/gh/bdhirsh/136/base 2022-03-17T19:07:02.9731246Z * [new branch] gh/bdhirsh/136/head -> origin/gh/bdhirsh/136/head 2022-03-17T19:07:02.9732664Z * [new branch] gh/bdhirsh/136/orig -> origin/gh/bdhirsh/136/orig 2022-03-17T19:07:02.9734732Z * [new branch] gh/bdhirsh/138/base -> origin/gh/bdhirsh/138/base 2022-03-17T19:07:02.9735882Z * [new branch] gh/bdhirsh/138/head -> origin/gh/bdhirsh/138/head 2022-03-17T19:07:02.9737268Z * [new branch] gh/bdhirsh/138/orig -> origin/gh/bdhirsh/138/orig 2022-03-17T19:07:02.9738850Z * [new branch] gh/bdhirsh/139/base -> origin/gh/bdhirsh/139/base 2022-03-17T19:07:02.9740058Z * [new branch] gh/bdhirsh/139/head -> origin/gh/bdhirsh/139/head 2022-03-17T19:07:02.9741480Z * [new branch] gh/bdhirsh/139/orig -> origin/gh/bdhirsh/139/orig 2022-03-17T19:07:02.9743521Z * [new branch] gh/bdhirsh/140/base -> origin/gh/bdhirsh/140/base 2022-03-17T19:07:02.9746489Z * [new branch] gh/bdhirsh/140/head -> origin/gh/bdhirsh/140/head 2022-03-17T19:07:02.9748514Z * [new branch] gh/bdhirsh/140/orig -> origin/gh/bdhirsh/140/orig 2022-03-17T19:07:02.9750422Z * [new branch] gh/bdhirsh/142/base -> origin/gh/bdhirsh/142/base 2022-03-17T19:07:02.9751792Z * [new branch] gh/bdhirsh/142/head -> origin/gh/bdhirsh/142/head 2022-03-17T19:07:02.9753569Z * [new branch] gh/bdhirsh/142/orig -> origin/gh/bdhirsh/142/orig 2022-03-17T19:07:02.9754716Z * [new branch] gh/bdhirsh/144/base -> origin/gh/bdhirsh/144/base 2022-03-17T19:07:02.9756077Z * [new branch] gh/bdhirsh/144/head -> origin/gh/bdhirsh/144/head 2022-03-17T19:07:02.9757561Z * [new branch] gh/bdhirsh/144/orig -> origin/gh/bdhirsh/144/orig 2022-03-17T19:07:02.9759641Z * [new branch] gh/bdhirsh/146/base -> origin/gh/bdhirsh/146/base 2022-03-17T19:07:02.9761023Z * [new branch] gh/bdhirsh/146/head -> origin/gh/bdhirsh/146/head 2022-03-17T19:07:02.9762498Z * [new branch] gh/bdhirsh/146/orig -> origin/gh/bdhirsh/146/orig 2022-03-17T19:07:02.9764602Z * [new branch] gh/bdhirsh/154/base -> origin/gh/bdhirsh/154/base 2022-03-17T19:07:02.9765881Z * [new branch] gh/bdhirsh/154/head -> origin/gh/bdhirsh/154/head 2022-03-17T19:07:02.9767293Z * [new branch] gh/bdhirsh/154/orig -> origin/gh/bdhirsh/154/orig 2022-03-17T19:07:02.9769274Z * [new branch] gh/bdhirsh/162/base -> origin/gh/bdhirsh/162/base 2022-03-17T19:07:02.9770627Z * [new branch] gh/bdhirsh/162/head -> origin/gh/bdhirsh/162/head 2022-03-17T19:07:02.9772100Z * [new branch] gh/bdhirsh/162/orig -> origin/gh/bdhirsh/162/orig 2022-03-17T19:07:02.9774176Z * [new branch] gh/bdhirsh/163/base -> origin/gh/bdhirsh/163/base 2022-03-17T19:07:02.9776073Z * [new branch] gh/bdhirsh/163/head -> origin/gh/bdhirsh/163/head 2022-03-17T19:07:02.9777106Z * [new branch] gh/bdhirsh/163/orig -> origin/gh/bdhirsh/163/orig 2022-03-17T19:07:02.9779425Z * [new branch] gh/bdhirsh/167/base -> origin/gh/bdhirsh/167/base 2022-03-17T19:07:02.9781183Z * [new branch] gh/bdhirsh/167/head -> origin/gh/bdhirsh/167/head 2022-03-17T19:07:02.9783283Z * [new branch] gh/bdhirsh/167/orig -> origin/gh/bdhirsh/167/orig 2022-03-17T19:07:02.9784781Z * [new branch] gh/bdhirsh/172/base -> origin/gh/bdhirsh/172/base 2022-03-17T19:07:02.9786069Z * [new branch] gh/bdhirsh/172/head -> origin/gh/bdhirsh/172/head 2022-03-17T19:07:02.9787397Z * [new branch] gh/bdhirsh/172/orig -> origin/gh/bdhirsh/172/orig 2022-03-17T19:07:02.9789428Z * [new branch] gh/bdhirsh/174/base -> origin/gh/bdhirsh/174/base 2022-03-17T19:07:02.9790548Z * [new branch] gh/bdhirsh/174/head -> origin/gh/bdhirsh/174/head 2022-03-17T19:07:02.9792798Z * [new branch] gh/bdhirsh/177/base -> origin/gh/bdhirsh/177/base 2022-03-17T19:07:02.9793933Z * [new branch] gh/bdhirsh/177/head -> origin/gh/bdhirsh/177/head 2022-03-17T19:07:02.9795368Z * [new branch] gh/bdhirsh/177/orig -> origin/gh/bdhirsh/177/orig 2022-03-17T19:07:02.9797344Z * [new branch] gh/bdhirsh/178/base -> origin/gh/bdhirsh/178/base 2022-03-17T19:07:02.9798399Z * [new branch] gh/bdhirsh/178/head -> origin/gh/bdhirsh/178/head 2022-03-17T19:07:02.9799668Z * [new branch] gh/bdhirsh/178/orig -> origin/gh/bdhirsh/178/orig 2022-03-17T19:07:02.9801652Z * [new branch] gh/bdhirsh/182/base -> origin/gh/bdhirsh/182/base 2022-03-17T19:07:02.9802793Z * [new branch] gh/bdhirsh/182/head -> origin/gh/bdhirsh/182/head 2022-03-17T19:07:02.9804189Z * [new branch] gh/bdhirsh/182/orig -> origin/gh/bdhirsh/182/orig 2022-03-17T19:07:02.9806280Z * [new branch] gh/bdhirsh/183/base -> origin/gh/bdhirsh/183/base 2022-03-17T19:07:02.9807335Z * [new branch] gh/bdhirsh/183/head -> origin/gh/bdhirsh/183/head 2022-03-17T19:07:02.9808797Z * [new branch] gh/bdhirsh/183/orig -> origin/gh/bdhirsh/183/orig 2022-03-17T19:07:02.9810948Z * [new branch] gh/bdhirsh/184/base -> origin/gh/bdhirsh/184/base 2022-03-17T19:07:02.9812143Z * [new branch] gh/bdhirsh/184/head -> origin/gh/bdhirsh/184/head 2022-03-17T19:07:02.9813771Z * [new branch] gh/bdhirsh/184/orig -> origin/gh/bdhirsh/184/orig 2022-03-17T19:07:02.9816235Z * [new branch] gh/bdhirsh/185/base -> origin/gh/bdhirsh/185/base 2022-03-17T19:07:02.9817359Z * [new branch] gh/bdhirsh/185/head -> origin/gh/bdhirsh/185/head 2022-03-17T19:07:02.9818666Z * [new branch] gh/bdhirsh/185/orig -> origin/gh/bdhirsh/185/orig 2022-03-17T19:07:02.9820550Z * [new branch] gh/bdhirsh/186/base -> origin/gh/bdhirsh/186/base 2022-03-17T19:07:02.9821742Z * [new branch] gh/bdhirsh/186/head -> origin/gh/bdhirsh/186/head 2022-03-17T19:07:02.9823477Z * [new branch] gh/bdhirsh/186/orig -> origin/gh/bdhirsh/186/orig 2022-03-17T19:07:02.9824651Z * [new branch] gh/bdhirsh/187/base -> origin/gh/bdhirsh/187/base 2022-03-17T19:07:02.9825989Z * [new branch] gh/bdhirsh/187/head -> origin/gh/bdhirsh/187/head 2022-03-17T19:07:02.9827329Z * [new branch] gh/bdhirsh/187/orig -> origin/gh/bdhirsh/187/orig 2022-03-17T19:07:02.9829381Z * [new branch] gh/bdhirsh/25/base -> origin/gh/bdhirsh/25/base 2022-03-17T19:07:02.9831621Z * [new branch] gh/bdhirsh/25/head -> origin/gh/bdhirsh/25/head 2022-03-17T19:07:02.9832823Z * [new branch] gh/bdhirsh/25/orig -> origin/gh/bdhirsh/25/orig 2022-03-17T19:07:02.9834829Z * [new branch] gh/bdhirsh/27/base -> origin/gh/bdhirsh/27/base 2022-03-17T19:07:02.9836126Z * [new branch] gh/bdhirsh/27/head -> origin/gh/bdhirsh/27/head 2022-03-17T19:07:02.9837473Z * [new branch] gh/bdhirsh/27/orig -> origin/gh/bdhirsh/27/orig 2022-03-17T19:07:02.9839504Z * [new branch] gh/bdhirsh/29/base -> origin/gh/bdhirsh/29/base 2022-03-17T19:07:02.9841399Z * [new branch] gh/bdhirsh/29/head -> origin/gh/bdhirsh/29/head 2022-03-17T19:07:02.9842579Z * [new branch] gh/bdhirsh/29/orig -> origin/gh/bdhirsh/29/orig 2022-03-17T19:07:02.9845343Z * [new branch] gh/bdhirsh/35/base -> origin/gh/bdhirsh/35/base 2022-03-17T19:07:02.9846628Z * [new branch] gh/bdhirsh/35/head -> origin/gh/bdhirsh/35/head 2022-03-17T19:07:02.9847988Z * [new branch] gh/bdhirsh/35/orig -> origin/gh/bdhirsh/35/orig 2022-03-17T19:07:02.9849962Z * [new branch] gh/bdhirsh/36/base -> origin/gh/bdhirsh/36/base 2022-03-17T19:07:02.9851028Z * [new branch] gh/bdhirsh/36/head -> origin/gh/bdhirsh/36/head 2022-03-17T19:07:02.9853128Z * [new branch] gh/bdhirsh/36/orig -> origin/gh/bdhirsh/36/orig 2022-03-17T19:07:02.9854410Z * [new branch] gh/bdhirsh/37/base -> origin/gh/bdhirsh/37/base 2022-03-17T19:07:02.9856021Z * [new branch] gh/bdhirsh/37/head -> origin/gh/bdhirsh/37/head 2022-03-17T19:07:02.9857279Z * [new branch] gh/bdhirsh/37/orig -> origin/gh/bdhirsh/37/orig 2022-03-17T19:07:02.9859594Z * [new branch] gh/bdhirsh/38/base -> origin/gh/bdhirsh/38/base 2022-03-17T19:07:02.9861173Z * [new branch] gh/bdhirsh/38/head -> origin/gh/bdhirsh/38/head 2022-03-17T19:07:02.9862352Z * [new branch] gh/bdhirsh/38/orig -> origin/gh/bdhirsh/38/orig 2022-03-17T19:07:02.9864335Z * [new branch] gh/bdhirsh/39/base -> origin/gh/bdhirsh/39/base 2022-03-17T19:07:02.9865567Z * [new branch] gh/bdhirsh/39/head -> origin/gh/bdhirsh/39/head 2022-03-17T19:07:02.9866910Z * [new branch] gh/bdhirsh/39/orig -> origin/gh/bdhirsh/39/orig 2022-03-17T19:07:02.9868996Z * [new branch] gh/bdhirsh/41/base -> origin/gh/bdhirsh/41/base 2022-03-17T19:07:02.9871248Z * [new branch] gh/bdhirsh/41/head -> origin/gh/bdhirsh/41/head 2022-03-17T19:07:02.9872444Z * [new branch] gh/bdhirsh/41/orig -> origin/gh/bdhirsh/41/orig 2022-03-17T19:07:02.9874749Z * [new branch] gh/bdhirsh/42/base -> origin/gh/bdhirsh/42/base 2022-03-17T19:07:02.9875834Z * [new branch] gh/bdhirsh/42/head -> origin/gh/bdhirsh/42/head 2022-03-17T19:07:02.9877218Z * [new branch] gh/bdhirsh/42/orig -> origin/gh/bdhirsh/42/orig 2022-03-17T19:07:02.9879151Z * [new branch] gh/bdhirsh/43/base -> origin/gh/bdhirsh/43/base 2022-03-17T19:07:02.9880162Z * [new branch] gh/bdhirsh/43/head -> origin/gh/bdhirsh/43/head 2022-03-17T19:07:02.9882368Z * [new branch] gh/bdhirsh/43/orig -> origin/gh/bdhirsh/43/orig 2022-03-17T19:07:02.9883652Z * [new branch] gh/bdhirsh/44/base -> origin/gh/bdhirsh/44/base 2022-03-17T19:07:02.9885276Z * [new branch] gh/bdhirsh/44/head -> origin/gh/bdhirsh/44/head 2022-03-17T19:07:02.9886468Z * [new branch] gh/bdhirsh/44/orig -> origin/gh/bdhirsh/44/orig 2022-03-17T19:07:02.9888332Z * [new branch] gh/bdhirsh/45/base -> origin/gh/bdhirsh/45/base 2022-03-17T19:07:02.9889381Z * [new branch] gh/bdhirsh/45/head -> origin/gh/bdhirsh/45/head 2022-03-17T19:07:02.9890788Z * [new branch] gh/bdhirsh/45/orig -> origin/gh/bdhirsh/45/orig 2022-03-17T19:07:02.9892809Z * [new branch] gh/bdhirsh/46/base -> origin/gh/bdhirsh/46/base 2022-03-17T19:07:02.9893864Z * [new branch] gh/bdhirsh/46/head -> origin/gh/bdhirsh/46/head 2022-03-17T19:07:02.9895193Z * [new branch] gh/bdhirsh/46/orig -> origin/gh/bdhirsh/46/orig 2022-03-17T19:07:02.9897135Z * [new branch] gh/bdhirsh/47/base -> origin/gh/bdhirsh/47/base 2022-03-17T19:07:02.9898149Z * [new branch] gh/bdhirsh/47/head -> origin/gh/bdhirsh/47/head 2022-03-17T19:07:02.9899532Z * [new branch] gh/bdhirsh/47/orig -> origin/gh/bdhirsh/47/orig 2022-03-17T19:07:02.9902204Z * [new branch] gh/bdhirsh/48/base -> origin/gh/bdhirsh/48/base 2022-03-17T19:07:02.9903288Z * [new branch] gh/bdhirsh/48/head -> origin/gh/bdhirsh/48/head 2022-03-17T19:07:02.9904629Z * [new branch] gh/bdhirsh/48/orig -> origin/gh/bdhirsh/48/orig 2022-03-17T19:07:02.9906648Z * [new branch] gh/bdhirsh/55/base -> origin/gh/bdhirsh/55/base 2022-03-17T19:07:02.9907789Z * [new branch] gh/bdhirsh/55/head -> origin/gh/bdhirsh/55/head 2022-03-17T19:07:02.9909868Z * [new branch] gh/bdhirsh/6/base -> origin/gh/bdhirsh/6/base 2022-03-17T19:07:02.9910929Z * [new branch] gh/bdhirsh/6/head -> origin/gh/bdhirsh/6/head 2022-03-17T19:07:02.9912825Z * [new branch] gh/bdhirsh/6/orig -> origin/gh/bdhirsh/6/orig 2022-03-17T19:07:02.9914191Z * [new branch] gh/bdhirsh/63/base -> origin/gh/bdhirsh/63/base 2022-03-17T19:07:02.9915526Z * [new branch] gh/bdhirsh/63/head -> origin/gh/bdhirsh/63/head 2022-03-17T19:07:02.9916860Z * [new branch] gh/bdhirsh/63/orig -> origin/gh/bdhirsh/63/orig 2022-03-17T19:07:02.9918816Z * [new branch] gh/bdhirsh/71/base -> origin/gh/bdhirsh/71/base 2022-03-17T19:07:02.9919914Z * [new branch] gh/bdhirsh/71/head -> origin/gh/bdhirsh/71/head 2022-03-17T19:07:02.9921585Z * [new branch] gh/bdhirsh/71/orig -> origin/gh/bdhirsh/71/orig 2022-03-17T19:07:02.9923335Z * [new branch] gh/bdhirsh/81/base -> origin/gh/bdhirsh/81/base 2022-03-17T19:07:02.9924408Z * [new branch] gh/bdhirsh/81/head -> origin/gh/bdhirsh/81/head 2022-03-17T19:07:02.9925989Z * [new branch] gh/bdhirsh/81/orig -> origin/gh/bdhirsh/81/orig 2022-03-17T19:07:02.9927734Z * [new branch] gh/bdhirsh/9/base -> origin/gh/bdhirsh/9/base 2022-03-17T19:07:02.9928992Z * [new branch] gh/bdhirsh/9/head -> origin/gh/bdhirsh/9/head 2022-03-17T19:07:02.9930295Z * [new branch] gh/bdhirsh/9/orig -> origin/gh/bdhirsh/9/orig 2022-03-17T19:07:02.9933676Z * [new branch] gh/bdhirsh/96/base -> origin/gh/bdhirsh/96/base 2022-03-17T19:07:02.9935649Z * [new branch] gh/bdhirsh/96/head -> origin/gh/bdhirsh/96/head 2022-03-17T19:07:02.9936843Z * [new branch] gh/bdhirsh/96/orig -> origin/gh/bdhirsh/96/orig 2022-03-17T19:07:02.9939050Z * [new branch] gh/beauby/1/base -> origin/gh/beauby/1/base 2022-03-17T19:07:02.9940419Z * [new branch] gh/beauby/1/head -> origin/gh/beauby/1/head 2022-03-17T19:07:02.9942241Z * [new branch] gh/beauby/2/base -> origin/gh/beauby/2/base 2022-03-17T19:07:02.9943490Z * [new branch] gh/beauby/3/base -> origin/gh/beauby/3/base 2022-03-17T19:07:02.9945725Z * [new branch] gh/bertmaher/172/base -> origin/gh/bertmaher/172/base 2022-03-17T19:07:02.9946959Z * [new branch] gh/bertmaher/172/head -> origin/gh/bertmaher/172/head 2022-03-17T19:07:02.9948555Z * [new branch] gh/bertmaher/172/orig -> origin/gh/bertmaher/172/orig 2022-03-17T19:07:02.9950493Z * [new branch] gh/bertmaher/79/base -> origin/gh/bertmaher/79/base 2022-03-17T19:07:02.9951702Z * [new branch] gh/bertmaher/79/head -> origin/gh/bertmaher/79/head 2022-03-17T19:07:02.9953388Z * [new branch] gh/bertmaher/80/base -> origin/gh/bertmaher/80/base 2022-03-17T19:07:02.9954592Z * [new branch] gh/bertmaher/80/head -> origin/gh/bertmaher/80/head 2022-03-17T19:07:02.9956804Z * [new branch] gh/bhosmer/10/base -> origin/gh/bhosmer/10/base 2022-03-17T19:07:02.9958411Z * [new branch] gh/bhosmer/10/head -> origin/gh/bhosmer/10/head 2022-03-17T19:07:02.9959360Z * [new branch] gh/bhosmer/10/orig -> origin/gh/bhosmer/10/orig 2022-03-17T19:07:02.9961469Z * [new branch] gh/bhosmer/11/base -> origin/gh/bhosmer/11/base 2022-03-17T19:07:02.9962652Z * [new branch] gh/bhosmer/11/head -> origin/gh/bhosmer/11/head 2022-03-17T19:07:02.9964264Z * [new branch] gh/bhosmer/11/orig -> origin/gh/bhosmer/11/orig 2022-03-17T19:07:02.9965997Z * [new branch] gh/bhosmer/12/base -> origin/gh/bhosmer/12/base 2022-03-17T19:07:02.9969684Z * [new branch] gh/bhosmer/12/head -> origin/gh/bhosmer/12/head 2022-03-17T19:07:02.9970838Z * [new branch] gh/bhosmer/12/orig -> origin/gh/bhosmer/12/orig 2022-03-17T19:07:02.9972449Z * [new branch] gh/bhosmer/13/base -> origin/gh/bhosmer/13/base 2022-03-17T19:07:02.9973954Z * [new branch] gh/bhosmer/13/head -> origin/gh/bhosmer/13/head 2022-03-17T19:07:02.9975869Z * [new branch] gh/bhosmer/13/orig -> origin/gh/bhosmer/13/orig 2022-03-17T19:07:02.9976987Z * [new branch] gh/bhosmer/14/base -> origin/gh/bhosmer/14/base 2022-03-17T19:07:02.9978289Z * [new branch] gh/bhosmer/14/head -> origin/gh/bhosmer/14/head 2022-03-17T19:07:02.9979683Z * [new branch] gh/bhosmer/14/orig -> origin/gh/bhosmer/14/orig 2022-03-17T19:07:02.9982874Z * [new branch] gh/bhosmer/17/base -> origin/gh/bhosmer/17/base 2022-03-17T19:07:02.9984035Z * [new branch] gh/bhosmer/17/head -> origin/gh/bhosmer/17/head 2022-03-17T19:07:02.9986042Z * [new branch] gh/bhosmer/17/orig -> origin/gh/bhosmer/17/orig 2022-03-17T19:07:02.9987046Z * [new branch] gh/bhosmer/18/base -> origin/gh/bhosmer/18/base 2022-03-17T19:07:02.9989442Z * [new branch] gh/bhosmer/18/head -> origin/gh/bhosmer/18/head 2022-03-17T19:07:02.9990924Z * [new branch] gh/bhosmer/18/orig -> origin/gh/bhosmer/18/orig 2022-03-17T19:07:02.9992340Z * [new branch] gh/bhosmer/19/base -> origin/gh/bhosmer/19/base 2022-03-17T19:07:02.9993750Z * [new branch] gh/bhosmer/19/head -> origin/gh/bhosmer/19/head 2022-03-17T19:07:02.9995115Z * [new branch] gh/bhosmer/19/orig -> origin/gh/bhosmer/19/orig 2022-03-17T19:07:02.9997484Z * [new branch] gh/bhosmer/2/base -> origin/gh/bhosmer/2/base 2022-03-17T19:07:02.9998677Z * [new branch] gh/bhosmer/2/head -> origin/gh/bhosmer/2/head 2022-03-17T19:07:03.0000041Z * [new branch] gh/bhosmer/2/orig -> origin/gh/bhosmer/2/orig 2022-03-17T19:07:03.0002205Z * [new branch] gh/bhosmer/21/base -> origin/gh/bhosmer/21/base 2022-03-17T19:07:03.0003201Z * [new branch] gh/bhosmer/21/head -> origin/gh/bhosmer/21/head 2022-03-17T19:07:03.0005018Z * [new branch] gh/bhosmer/21/orig -> origin/gh/bhosmer/21/orig 2022-03-17T19:07:03.0006410Z * [new branch] gh/bhosmer/24/base -> origin/gh/bhosmer/24/base 2022-03-17T19:07:03.0007781Z * [new branch] gh/bhosmer/24/head -> origin/gh/bhosmer/24/head 2022-03-17T19:07:03.0009131Z * [new branch] gh/bhosmer/24/orig -> origin/gh/bhosmer/24/orig 2022-03-17T19:07:03.0011213Z * [new branch] gh/bhosmer/25/base -> origin/gh/bhosmer/25/base 2022-03-17T19:07:03.0012211Z * [new branch] gh/bhosmer/25/head -> origin/gh/bhosmer/25/head 2022-03-17T19:07:03.0013891Z * [new branch] gh/bhosmer/25/orig -> origin/gh/bhosmer/25/orig 2022-03-17T19:07:03.0015154Z * [new branch] gh/bhosmer/3/base -> origin/gh/bhosmer/3/base 2022-03-17T19:07:03.0016760Z * [new branch] gh/bhosmer/3/head -> origin/gh/bhosmer/3/head 2022-03-17T19:07:03.0018093Z * [new branch] gh/bhosmer/3/orig -> origin/gh/bhosmer/3/orig 2022-03-17T19:07:03.0020062Z * [new branch] gh/bhosmer/33/base -> origin/gh/bhosmer/33/base 2022-03-17T19:07:03.0021157Z * [new branch] gh/bhosmer/33/head -> origin/gh/bhosmer/33/head 2022-03-17T19:07:03.0023088Z * [new branch] gh/bhosmer/33/orig -> origin/gh/bhosmer/33/orig 2022-03-17T19:07:03.0024453Z * [new branch] gh/bhosmer/42/base -> origin/gh/bhosmer/42/base 2022-03-17T19:07:03.0025812Z * [new branch] gh/bhosmer/42/head -> origin/gh/bhosmer/42/head 2022-03-17T19:07:03.0027179Z * [new branch] gh/bhosmer/42/orig -> origin/gh/bhosmer/42/orig 2022-03-17T19:07:03.0030066Z * [new branch] gh/bhosmer/43/base -> origin/gh/bhosmer/43/base 2022-03-17T19:07:03.0031124Z * [new branch] gh/bhosmer/43/head -> origin/gh/bhosmer/43/head 2022-03-17T19:07:03.0032466Z * [new branch] gh/bhosmer/43/orig -> origin/gh/bhosmer/43/orig 2022-03-17T19:07:03.0034562Z * [new branch] gh/bhosmer/6/base -> origin/gh/bhosmer/6/base 2022-03-17T19:07:03.0036326Z * [new branch] gh/bhosmer/6/head -> origin/gh/bhosmer/6/head 2022-03-17T19:07:03.0037615Z * [new branch] gh/bhosmer/6/orig -> origin/gh/bhosmer/6/orig 2022-03-17T19:07:03.0039312Z * [new branch] gh/bhosmer/7/base -> origin/gh/bhosmer/7/base 2022-03-17T19:07:03.0040263Z * [new branch] gh/bhosmer/7/head -> origin/gh/bhosmer/7/head 2022-03-17T19:07:03.0041734Z * [new branch] gh/bhosmer/7/orig -> origin/gh/bhosmer/7/orig 2022-03-17T19:07:03.0043615Z * [new branch] gh/bhosmer/8/base -> origin/gh/bhosmer/8/base 2022-03-17T19:07:03.0044661Z * [new branch] gh/bhosmer/8/head -> origin/gh/bhosmer/8/head 2022-03-17T19:07:03.0046042Z * [new branch] gh/bhosmer/8/orig -> origin/gh/bhosmer/8/orig 2022-03-17T19:07:03.0047746Z * [new branch] gh/bhosmer/9/base -> origin/gh/bhosmer/9/base 2022-03-17T19:07:03.0048963Z * [new branch] gh/bhosmer/9/head -> origin/gh/bhosmer/9/head 2022-03-17T19:07:03.0050695Z * [new branch] gh/bhosmer/9/orig -> origin/gh/bhosmer/9/orig 2022-03-17T19:07:03.0052659Z * [new branch] gh/blefaudeux/4/base -> origin/gh/blefaudeux/4/base 2022-03-17T19:07:03.0053754Z * [new branch] gh/blefaudeux/4/head -> origin/gh/blefaudeux/4/head 2022-03-17T19:07:03.0055087Z * [new branch] gh/blefaudeux/4/orig -> origin/gh/blefaudeux/4/orig 2022-03-17T19:07:03.0057389Z * [new branch] gh/bowangbj/1/base -> origin/gh/bowangbj/1/base 2022-03-17T19:07:03.0058538Z * [new branch] gh/bowangbj/1/orig -> origin/gh/bowangbj/1/orig 2022-03-17T19:07:03.0060717Z * [new branch] gh/bowangbj/11/base -> origin/gh/bowangbj/11/base 2022-03-17T19:07:03.0062146Z * [new branch] gh/bowangbj/11/head -> origin/gh/bowangbj/11/head 2022-03-17T19:07:03.0063657Z * [new branch] gh/bowangbj/11/orig -> origin/gh/bowangbj/11/orig 2022-03-17T19:07:03.0065914Z * [new branch] gh/bowangbj/13/base -> origin/gh/bowangbj/13/base 2022-03-17T19:07:03.0067860Z * [new branch] gh/bowangbj/13/head -> origin/gh/bowangbj/13/head 2022-03-17T19:07:03.0068968Z * [new branch] gh/bowangbj/13/orig -> origin/gh/bowangbj/13/orig 2022-03-17T19:07:03.0071081Z * [new branch] gh/bowangbj/2/base -> origin/gh/bowangbj/2/base 2022-03-17T19:07:03.0072082Z * [new branch] gh/bowangbj/2/orig -> origin/gh/bowangbj/2/orig 2022-03-17T19:07:03.0074155Z * [new branch] gh/bowangbj/3/base -> origin/gh/bowangbj/3/base 2022-03-17T19:07:03.0075301Z * [new branch] gh/bowangbj/3/head -> origin/gh/bowangbj/3/head 2022-03-17T19:07:03.0076903Z * [new branch] gh/bowangbj/3/orig -> origin/gh/bowangbj/3/orig 2022-03-17T19:07:03.0078502Z * [new branch] gh/bowangbj/4/base -> origin/gh/bowangbj/4/base 2022-03-17T19:07:03.0079766Z * [new branch] gh/bowangbj/4/orig -> origin/gh/bowangbj/4/orig 2022-03-17T19:07:03.0081817Z * [new branch] gh/bowangbj/9/base -> origin/gh/bowangbj/9/base 2022-03-17T19:07:03.0082847Z * [new branch] gh/bowangbj/9/head -> origin/gh/bowangbj/9/head 2022-03-17T19:07:03.0084217Z * [new branch] gh/bowangbj/9/orig -> origin/gh/bowangbj/9/orig 2022-03-17T19:07:03.0086571Z * [new branch] gh/bwasti/10/base -> origin/gh/bwasti/10/base 2022-03-17T19:07:03.0087783Z * [new branch] gh/bwasti/10/head -> origin/gh/bwasti/10/head 2022-03-17T19:07:03.0089748Z * [new branch] gh/bwasti/13/base -> origin/gh/bwasti/13/base 2022-03-17T19:07:03.0090816Z * [new branch] gh/bwasti/13/head -> origin/gh/bwasti/13/head 2022-03-17T19:07:03.0092554Z * [new branch] gh/bwasti/14/base -> origin/gh/bwasti/14/base 2022-03-17T19:07:03.0093653Z * [new branch] gh/bwasti/14/head -> origin/gh/bwasti/14/head 2022-03-17T19:07:03.0095527Z * [new branch] gh/bwasti/16/base -> origin/gh/bwasti/16/base 2022-03-17T19:07:03.0096481Z * [new branch] gh/bwasti/16/head -> origin/gh/bwasti/16/head 2022-03-17T19:07:03.0098498Z * [new branch] gh/bwasti/17/base -> origin/gh/bwasti/17/base 2022-03-17T19:07:03.0099509Z * [new branch] gh/bwasti/17/head -> origin/gh/bwasti/17/head 2022-03-17T19:07:03.0101639Z * [new branch] gh/bwasti/25/base -> origin/gh/bwasti/25/base 2022-03-17T19:07:03.0102962Z * [new branch] gh/bwasti/25/head -> origin/gh/bwasti/25/head 2022-03-17T19:07:03.0105056Z * [new branch] gh/bwasti/26/base -> origin/gh/bwasti/26/base 2022-03-17T19:07:03.0106009Z * [new branch] gh/bwasti/26/head -> origin/gh/bwasti/26/head 2022-03-17T19:07:03.0108069Z * [new branch] gh/bwasti/28/base -> origin/gh/bwasti/28/base 2022-03-17T19:07:03.0109089Z * [new branch] gh/bwasti/28/head -> origin/gh/bwasti/28/head 2022-03-17T19:07:03.0110701Z * [new branch] gh/bwasti/31/base -> origin/gh/bwasti/31/base 2022-03-17T19:07:03.0111969Z * [new branch] gh/bwasti/31/head -> origin/gh/bwasti/31/head 2022-03-17T19:07:03.0113346Z * [new branch] gh/bwasti/31/orig -> origin/gh/bwasti/31/orig 2022-03-17T19:07:03.0115868Z * [new branch] gh/bwasti/33/base -> origin/gh/bwasti/33/base 2022-03-17T19:07:03.0117068Z * [new branch] gh/bwasti/33/head -> origin/gh/bwasti/33/head 2022-03-17T19:07:03.0118382Z * [new branch] gh/bwasti/33/orig -> origin/gh/bwasti/33/orig 2022-03-17T19:07:03.0120626Z * [new branch] gh/bwasti/35/base -> origin/gh/bwasti/35/base 2022-03-17T19:07:03.0121798Z * [new branch] gh/bwasti/35/head -> origin/gh/bwasti/35/head 2022-03-17T19:07:03.0123401Z * [new branch] gh/bwasti/35/orig -> origin/gh/bwasti/35/orig 2022-03-17T19:07:03.0124665Z * [new branch] gh/bwasti/36/base -> origin/gh/bwasti/36/base 2022-03-17T19:07:03.0125987Z * [new branch] gh/bwasti/36/head -> origin/gh/bwasti/36/head 2022-03-17T19:07:03.0127338Z * [new branch] gh/bwasti/36/orig -> origin/gh/bwasti/36/orig 2022-03-17T19:07:03.0129275Z * [new branch] gh/bwasti/40/base -> origin/gh/bwasti/40/base 2022-03-17T19:07:03.0130349Z * [new branch] gh/bwasti/40/head -> origin/gh/bwasti/40/head 2022-03-17T19:07:03.0132254Z * [new branch] gh/bwasti/41/base -> origin/gh/bwasti/41/base 2022-03-17T19:07:03.0133294Z * [new branch] gh/bwasti/41/head -> origin/gh/bwasti/41/head 2022-03-17T19:07:03.0135251Z * [new branch] gh/bwasti/41/orig -> origin/gh/bwasti/41/orig 2022-03-17T19:07:03.0136444Z * [new branch] gh/bwasti/49/base -> origin/gh/bwasti/49/base 2022-03-17T19:07:03.0137799Z * [new branch] gh/bwasti/49/head -> origin/gh/bwasti/49/head 2022-03-17T19:07:03.0139104Z * [new branch] gh/bwasti/49/orig -> origin/gh/bwasti/49/orig 2022-03-17T19:07:03.0141410Z * [new branch] gh/bwasti/51/base -> origin/gh/bwasti/51/base 2022-03-17T19:07:03.0142662Z * [new branch] gh/bwasti/51/head -> origin/gh/bwasti/51/head 2022-03-17T19:07:03.0144135Z * [new branch] gh/bwasti/51/orig -> origin/gh/bwasti/51/orig 2022-03-17T19:07:03.0146059Z * [new branch] gh/bwasti/9/base -> origin/gh/bwasti/9/base 2022-03-17T19:07:03.0146778Z * [new branch] gh/bwasti/9/head -> origin/gh/bwasti/9/head 2022-03-17T19:07:03.0149419Z * [new branch] gh/bzinodev/1/base -> origin/gh/bzinodev/1/base 2022-03-17T19:07:03.0150427Z * [new branch] gh/bzinodev/1/head -> origin/gh/bzinodev/1/head 2022-03-17T19:07:03.0151830Z * [new branch] gh/bzinodev/2/base -> origin/gh/bzinodev/2/base 2022-03-17T19:07:03.0153133Z * [new branch] gh/bzinodev/2/head -> origin/gh/bzinodev/2/head 2022-03-17T19:07:03.0155145Z * [new branch] gh/bzinodev/20/base -> origin/gh/bzinodev/20/base 2022-03-17T19:07:03.0156203Z * [new branch] gh/bzinodev/20/head -> origin/gh/bzinodev/20/head 2022-03-17T19:07:03.0158120Z * [new branch] gh/bzinodev/20/orig -> origin/gh/bzinodev/20/orig 2022-03-17T19:07:03.0159503Z * [new branch] gh/bzinodev/6/base -> origin/gh/bzinodev/6/base 2022-03-17T19:07:03.0161000Z * [new branch] gh/bzinodev/6/head -> origin/gh/bzinodev/6/head 2022-03-17T19:07:03.0162362Z * [new branch] gh/bzinodev/6/orig -> origin/gh/bzinodev/6/orig 2022-03-17T19:07:03.0164317Z * [new branch] gh/bzinodev/7/base -> origin/gh/bzinodev/7/base 2022-03-17T19:07:03.0165451Z * [new branch] gh/bzinodev/7/head -> origin/gh/bzinodev/7/head 2022-03-17T19:07:03.0166778Z * [new branch] gh/bzinodev/7/orig -> origin/gh/bzinodev/7/orig 2022-03-17T19:07:03.0169475Z * [new branch] gh/cbalioglu/24/base -> origin/gh/cbalioglu/24/base 2022-03-17T19:07:03.0170689Z * [new branch] gh/cbalioglu/24/orig -> origin/gh/cbalioglu/24/orig 2022-03-17T19:07:03.0172749Z * [new branch] gh/cbalioglu/25/base -> origin/gh/cbalioglu/25/base 2022-03-17T19:07:03.0173855Z * [new branch] gh/cbalioglu/25/orig -> origin/gh/cbalioglu/25/orig 2022-03-17T19:07:03.0176185Z * [new branch] gh/cbalioglu/30/base -> origin/gh/cbalioglu/30/base 2022-03-17T19:07:03.0177359Z * [new branch] gh/cbalioglu/30/orig -> origin/gh/cbalioglu/30/orig 2022-03-17T19:07:03.0179319Z * [new branch] gh/cbalioglu/31/base -> origin/gh/cbalioglu/31/base 2022-03-17T19:07:03.0180576Z * [new branch] gh/cbalioglu/31/orig -> origin/gh/cbalioglu/31/orig 2022-03-17T19:07:03.0186912Z * [new branch] gh/cbalioglu/37/base -> origin/gh/cbalioglu/37/base 2022-03-17T19:07:03.0187256Z * [new branch] gh/cbalioglu/37/head -> origin/gh/cbalioglu/37/head 2022-03-17T19:07:03.0187541Z * [new branch] gh/cbalioglu/38/base -> origin/gh/cbalioglu/38/base 2022-03-17T19:07:03.0191315Z * [new branch] gh/cbalioglu/38/head -> origin/gh/cbalioglu/38/head 2022-03-17T19:07:03.0191625Z * [new branch] gh/cbalioglu/39/base -> origin/gh/cbalioglu/39/base 2022-03-17T19:07:03.0191904Z * [new branch] gh/cbalioglu/39/head -> origin/gh/cbalioglu/39/head 2022-03-17T19:07:03.0192155Z * [new branch] gh/cbalioglu/40/base -> origin/gh/cbalioglu/40/base 2022-03-17T19:07:03.0192416Z * [new branch] gh/cbalioglu/40/head -> origin/gh/cbalioglu/40/head 2022-03-17T19:07:03.0193716Z * [new branch] gh/cbalioglu/46/base -> origin/gh/cbalioglu/46/base 2022-03-17T19:07:03.0195277Z * [new branch] gh/cbalioglu/46/orig -> origin/gh/cbalioglu/46/orig 2022-03-17T19:07:03.0197136Z * [new branch] gh/cbalioglu/48/base -> origin/gh/cbalioglu/48/base 2022-03-17T19:07:03.0198506Z * [new branch] gh/cbalioglu/48/orig -> origin/gh/cbalioglu/48/orig 2022-03-17T19:07:03.0200300Z * [new branch] gh/cbalioglu/49/base -> origin/gh/cbalioglu/49/base 2022-03-17T19:07:03.0201793Z * [new branch] gh/cbalioglu/49/orig -> origin/gh/cbalioglu/49/orig 2022-03-17T19:07:03.0203501Z * [new branch] gh/cbalioglu/56/base -> origin/gh/cbalioglu/56/base 2022-03-17T19:07:03.0204812Z * [new branch] gh/cbalioglu/56/orig -> origin/gh/cbalioglu/56/orig 2022-03-17T19:07:03.0206524Z * [new branch] gh/cbalioglu/57/base -> origin/gh/cbalioglu/57/base 2022-03-17T19:07:03.0207899Z * [new branch] gh/cbalioglu/57/orig -> origin/gh/cbalioglu/57/orig 2022-03-17T19:07:03.0209572Z * [new branch] gh/cbalioglu/58/base -> origin/gh/cbalioglu/58/base 2022-03-17T19:07:03.0211134Z * [new branch] gh/cbalioglu/58/orig -> origin/gh/cbalioglu/58/orig 2022-03-17T19:07:03.0213847Z * [new branch] gh/cccclai/10/base -> origin/gh/cccclai/10/base 2022-03-17T19:07:03.0214285Z * [new branch] gh/cccclai/10/head -> origin/gh/cccclai/10/head 2022-03-17T19:07:03.0216133Z * [new branch] gh/cccclai/10/orig -> origin/gh/cccclai/10/orig 2022-03-17T19:07:03.0217350Z * [new branch] gh/cccclai/101/base -> origin/gh/cccclai/101/base 2022-03-17T19:07:03.0218984Z * [new branch] gh/cccclai/101/head -> origin/gh/cccclai/101/head 2022-03-17T19:07:03.0219917Z * [new branch] gh/cccclai/101/orig -> origin/gh/cccclai/101/orig 2022-03-17T19:07:03.0222177Z * [new branch] gh/cccclai/104/base -> origin/gh/cccclai/104/base 2022-03-17T19:07:03.0224096Z * [new branch] gh/cccclai/104/head -> origin/gh/cccclai/104/head 2022-03-17T19:07:03.0225259Z * [new branch] gh/cccclai/104/orig -> origin/gh/cccclai/104/orig 2022-03-17T19:07:03.0227250Z * [new branch] gh/cccclai/106/base -> origin/gh/cccclai/106/base 2022-03-17T19:07:03.0228229Z * [new branch] gh/cccclai/106/head -> origin/gh/cccclai/106/head 2022-03-17T19:07:03.0230116Z * [new branch] gh/cccclai/106/orig -> origin/gh/cccclai/106/orig 2022-03-17T19:07:03.0232625Z * [new branch] gh/cccclai/109/base -> origin/gh/cccclai/109/base 2022-03-17T19:07:03.0234017Z * [new branch] gh/cccclai/109/head -> origin/gh/cccclai/109/head 2022-03-17T19:07:03.0235753Z * [new branch] gh/cccclai/109/orig -> origin/gh/cccclai/109/orig 2022-03-17T19:07:03.0237609Z * [new branch] gh/cccclai/110/base -> origin/gh/cccclai/110/base 2022-03-17T19:07:03.0238748Z * [new branch] gh/cccclai/110/head -> origin/gh/cccclai/110/head 2022-03-17T19:07:03.0240333Z * [new branch] gh/cccclai/110/orig -> origin/gh/cccclai/110/orig 2022-03-17T19:07:03.0242546Z * [new branch] gh/cccclai/111/base -> origin/gh/cccclai/111/base 2022-03-17T19:07:03.0243666Z * [new branch] gh/cccclai/111/head -> origin/gh/cccclai/111/head 2022-03-17T19:07:03.0245253Z * [new branch] gh/cccclai/111/orig -> origin/gh/cccclai/111/orig 2022-03-17T19:07:03.0246889Z * [new branch] gh/cccclai/112/base -> origin/gh/cccclai/112/base 2022-03-17T19:07:03.0248087Z * [new branch] gh/cccclai/112/head -> origin/gh/cccclai/112/head 2022-03-17T19:07:03.0249776Z * [new branch] gh/cccclai/112/orig -> origin/gh/cccclai/112/orig 2022-03-17T19:07:03.0251813Z * [new branch] gh/cccclai/113/base -> origin/gh/cccclai/113/base 2022-03-17T19:07:03.0253972Z * [new branch] gh/cccclai/113/head -> origin/gh/cccclai/113/head 2022-03-17T19:07:03.0255236Z * [new branch] gh/cccclai/113/orig -> origin/gh/cccclai/113/orig 2022-03-17T19:07:03.0257267Z * [new branch] gh/cccclai/116/base -> origin/gh/cccclai/116/base 2022-03-17T19:07:03.0258909Z * [new branch] gh/cccclai/116/head -> origin/gh/cccclai/116/head 2022-03-17T19:07:03.0260101Z * [new branch] gh/cccclai/116/orig -> origin/gh/cccclai/116/orig 2022-03-17T19:07:03.0262340Z * [new branch] gh/cccclai/121/base -> origin/gh/cccclai/121/base 2022-03-17T19:07:03.0263464Z * [new branch] gh/cccclai/121/head -> origin/gh/cccclai/121/head 2022-03-17T19:07:03.0265077Z * [new branch] gh/cccclai/121/orig -> origin/gh/cccclai/121/orig 2022-03-17T19:07:03.0267202Z * [new branch] gh/cccclai/122/base -> origin/gh/cccclai/122/base 2022-03-17T19:07:03.0268395Z * [new branch] gh/cccclai/122/head -> origin/gh/cccclai/122/head 2022-03-17T19:07:03.0270170Z * [new branch] gh/cccclai/122/orig -> origin/gh/cccclai/122/orig 2022-03-17T19:07:03.0272179Z * [new branch] gh/cccclai/128/base -> origin/gh/cccclai/128/base 2022-03-17T19:07:03.0273867Z * [new branch] gh/cccclai/128/head -> origin/gh/cccclai/128/head 2022-03-17T19:07:03.0275137Z * [new branch] gh/cccclai/128/orig -> origin/gh/cccclai/128/orig 2022-03-17T19:07:03.0277272Z * [new branch] gh/cccclai/130/base -> origin/gh/cccclai/130/base 2022-03-17T19:07:03.0278601Z * [new branch] gh/cccclai/130/head -> origin/gh/cccclai/130/head 2022-03-17T19:07:03.0280212Z * [new branch] gh/cccclai/130/orig -> origin/gh/cccclai/130/orig 2022-03-17T19:07:03.0282111Z * [new branch] gh/cccclai/136/base -> origin/gh/cccclai/136/base 2022-03-17T19:07:03.0283346Z * [new branch] gh/cccclai/136/head -> origin/gh/cccclai/136/head 2022-03-17T19:07:03.0285585Z * [new branch] gh/cccclai/136/orig -> origin/gh/cccclai/136/orig 2022-03-17T19:07:03.0287526Z * [new branch] gh/cccclai/139/base -> origin/gh/cccclai/139/base 2022-03-17T19:07:03.0288600Z * [new branch] gh/cccclai/139/head -> origin/gh/cccclai/139/head 2022-03-17T19:07:03.0290161Z * [new branch] gh/cccclai/139/orig -> origin/gh/cccclai/139/orig 2022-03-17T19:07:03.0291581Z * [new branch] gh/cccclai/14/base -> origin/gh/cccclai/14/base 2022-03-17T19:07:03.0293410Z * [new branch] gh/cccclai/14/head -> origin/gh/cccclai/14/head 2022-03-17T19:07:03.0295165Z * [new branch] gh/cccclai/141/base -> origin/gh/cccclai/141/base 2022-03-17T19:07:03.0296544Z * [new branch] gh/cccclai/141/head -> origin/gh/cccclai/141/head 2022-03-17T19:07:03.0297928Z * [new branch] gh/cccclai/141/orig -> origin/gh/cccclai/141/orig 2022-03-17T19:07:03.0300706Z * [new branch] gh/cccclai/146/base -> origin/gh/cccclai/146/base 2022-03-17T19:07:03.0302085Z * [new branch] gh/cccclai/146/head -> origin/gh/cccclai/146/head 2022-03-17T19:07:03.0303859Z * [new branch] gh/cccclai/147/base -> origin/gh/cccclai/147/base 2022-03-17T19:07:03.0304970Z * [new branch] gh/cccclai/147/head -> origin/gh/cccclai/147/head 2022-03-17T19:07:03.0307046Z * [new branch] gh/cccclai/149/base -> origin/gh/cccclai/149/base 2022-03-17T19:07:03.0308465Z * [new branch] gh/cccclai/149/head -> origin/gh/cccclai/149/head 2022-03-17T19:07:03.0310534Z * [new branch] gh/cccclai/149/orig -> origin/gh/cccclai/149/orig 2022-03-17T19:07:03.0312010Z * [new branch] gh/cccclai/15/base -> origin/gh/cccclai/15/base 2022-03-17T19:07:03.0314540Z * [new branch] gh/cccclai/15/head -> origin/gh/cccclai/15/head 2022-03-17T19:07:03.0315657Z * [new branch] gh/cccclai/15/orig -> origin/gh/cccclai/15/orig 2022-03-17T19:07:03.0317678Z * [new branch] gh/cccclai/154/base -> origin/gh/cccclai/154/base 2022-03-17T19:07:03.0318942Z * [new branch] gh/cccclai/154/head -> origin/gh/cccclai/154/head 2022-03-17T19:07:03.0320706Z * [new branch] gh/cccclai/155/base -> origin/gh/cccclai/155/base 2022-03-17T19:07:03.0321769Z * [new branch] gh/cccclai/155/head -> origin/gh/cccclai/155/head 2022-03-17T19:07:03.0331492Z * [new branch] gh/cccclai/155/orig -> origin/gh/cccclai/155/orig 2022-03-17T19:07:03.0333426Z * [new branch] gh/cccclai/156/base -> origin/gh/cccclai/156/base 2022-03-17T19:07:03.0334559Z * [new branch] gh/cccclai/156/head -> origin/gh/cccclai/156/head 2022-03-17T19:07:03.0336066Z * [new branch] gh/cccclai/156/orig -> origin/gh/cccclai/156/orig 2022-03-17T19:07:03.0338585Z * [new branch] gh/cccclai/157/base -> origin/gh/cccclai/157/base 2022-03-17T19:07:03.0343400Z * [new branch] gh/cccclai/157/head -> origin/gh/cccclai/157/head 2022-03-17T19:07:03.0345765Z * [new branch] gh/cccclai/157/orig -> origin/gh/cccclai/157/orig 2022-03-17T19:07:03.0348751Z * [new branch] gh/cccclai/160/base -> origin/gh/cccclai/160/base 2022-03-17T19:07:03.0350354Z * [new branch] gh/cccclai/160/head -> origin/gh/cccclai/160/head 2022-03-17T19:07:03.0353868Z * [new branch] gh/cccclai/20/base -> origin/gh/cccclai/20/base 2022-03-17T19:07:03.0355420Z * [new branch] gh/cccclai/20/head -> origin/gh/cccclai/20/head 2022-03-17T19:07:03.0358321Z * [new branch] gh/cccclai/20/orig -> origin/gh/cccclai/20/orig 2022-03-17T19:07:03.0361284Z * [new branch] gh/cccclai/26/base -> origin/gh/cccclai/26/base 2022-03-17T19:07:03.0363115Z * [new branch] gh/cccclai/26/head -> origin/gh/cccclai/26/head 2022-03-17T19:07:03.0365886Z * [new branch] gh/cccclai/26/orig -> origin/gh/cccclai/26/orig 2022-03-17T19:07:03.0368815Z * [new branch] gh/cccclai/29/base -> origin/gh/cccclai/29/base 2022-03-17T19:07:03.0370803Z * [new branch] gh/cccclai/29/head -> origin/gh/cccclai/29/head 2022-03-17T19:07:03.0373243Z * [new branch] gh/cccclai/29/orig -> origin/gh/cccclai/29/orig 2022-03-17T19:07:03.0377431Z * [new branch] gh/cccclai/32/base -> origin/gh/cccclai/32/base 2022-03-17T19:07:03.0378945Z * [new branch] gh/cccclai/32/head -> origin/gh/cccclai/32/head 2022-03-17T19:07:03.0382694Z * [new branch] gh/cccclai/33/base -> origin/gh/cccclai/33/base 2022-03-17T19:07:03.0384197Z * [new branch] gh/cccclai/33/head -> origin/gh/cccclai/33/head 2022-03-17T19:07:03.0387580Z * [new branch] gh/cccclai/38/base -> origin/gh/cccclai/38/base 2022-03-17T19:07:03.0389310Z * [new branch] gh/cccclai/38/head -> origin/gh/cccclai/38/head 2022-03-17T19:07:03.0392494Z * [new branch] gh/cccclai/38/orig -> origin/gh/cccclai/38/orig 2022-03-17T19:07:03.0395320Z * [new branch] gh/cccclai/40/base -> origin/gh/cccclai/40/base 2022-03-17T19:07:03.0399256Z * [new branch] gh/cccclai/40/head -> origin/gh/cccclai/40/head 2022-03-17T19:07:03.0400786Z * [new branch] gh/cccclai/40/orig -> origin/gh/cccclai/40/orig 2022-03-17T19:07:03.0403967Z * [new branch] gh/cccclai/46/base -> origin/gh/cccclai/46/base 2022-03-17T19:07:03.0405539Z * [new branch] gh/cccclai/46/head -> origin/gh/cccclai/46/head 2022-03-17T19:07:03.0409219Z * [new branch] gh/cccclai/49/base -> origin/gh/cccclai/49/base 2022-03-17T19:07:03.0410684Z * [new branch] gh/cccclai/49/head -> origin/gh/cccclai/49/head 2022-03-17T19:07:03.0413456Z * [new branch] gh/cccclai/49/orig -> origin/gh/cccclai/49/orig 2022-03-17T19:07:03.0416459Z * [new branch] gh/cccclai/65/base -> origin/gh/cccclai/65/base 2022-03-17T19:07:03.0418109Z * [new branch] gh/cccclai/65/head -> origin/gh/cccclai/65/head 2022-03-17T19:07:03.0421172Z * [new branch] gh/cccclai/65/orig -> origin/gh/cccclai/65/orig 2022-03-17T19:07:03.0424100Z * [new branch] gh/cccclai/71/base -> origin/gh/cccclai/71/base 2022-03-17T19:07:03.0425670Z * [new branch] gh/cccclai/71/head -> origin/gh/cccclai/71/head 2022-03-17T19:07:03.0428576Z * [new branch] gh/cccclai/71/orig -> origin/gh/cccclai/71/orig 2022-03-17T19:07:03.0431544Z * [new branch] gh/cccclai/72/base -> origin/gh/cccclai/72/base 2022-03-17T19:07:03.0433210Z * [new branch] gh/cccclai/72/head -> origin/gh/cccclai/72/head 2022-03-17T19:07:03.0436784Z * [new branch] gh/cccclai/72/orig -> origin/gh/cccclai/72/orig 2022-03-17T19:07:03.0439590Z * [new branch] gh/cccclai/78/base -> origin/gh/cccclai/78/base 2022-03-17T19:07:03.0442719Z * [new branch] gh/cccclai/78/head -> origin/gh/cccclai/78/head 2022-03-17T19:07:03.0445653Z * [new branch] gh/cccclai/78/orig -> origin/gh/cccclai/78/orig 2022-03-17T19:07:03.0448430Z * [new branch] gh/cccclai/83/base -> origin/gh/cccclai/83/base 2022-03-17T19:07:03.0450747Z * [new branch] gh/cccclai/83/head -> origin/gh/cccclai/83/head 2022-03-17T19:07:03.0453066Z * [new branch] gh/cccclai/83/orig -> origin/gh/cccclai/83/orig 2022-03-17T19:07:03.0456215Z * [new branch] gh/cccclai/85/base -> origin/gh/cccclai/85/base 2022-03-17T19:07:03.0457620Z * [new branch] gh/cccclai/85/head -> origin/gh/cccclai/85/head 2022-03-17T19:07:03.0460691Z * [new branch] gh/cccclai/85/orig -> origin/gh/cccclai/85/orig 2022-03-17T19:07:03.0463251Z * [new branch] gh/cccclai/86/base -> origin/gh/cccclai/86/base 2022-03-17T19:07:03.0465015Z * [new branch] gh/cccclai/86/head -> origin/gh/cccclai/86/head 2022-03-17T19:07:03.0467842Z * [new branch] gh/cccclai/86/orig -> origin/gh/cccclai/86/orig 2022-03-17T19:07:03.0470371Z * [new branch] gh/cccclai/87/base -> origin/gh/cccclai/87/base 2022-03-17T19:07:03.0472660Z * [new branch] gh/cccclai/87/head -> origin/gh/cccclai/87/head 2022-03-17T19:07:03.0474969Z * [new branch] gh/cccclai/87/orig -> origin/gh/cccclai/87/orig 2022-03-17T19:07:03.0478010Z * [new branch] gh/cccclai/89/base -> origin/gh/cccclai/89/base 2022-03-17T19:07:03.0479569Z * [new branch] gh/cccclai/89/head -> origin/gh/cccclai/89/head 2022-03-17T19:07:03.0482263Z * [new branch] gh/cccclai/89/orig -> origin/gh/cccclai/89/orig 2022-03-17T19:07:03.0485297Z * [new branch] gh/cccclai/92/base -> origin/gh/cccclai/92/base 2022-03-17T19:07:03.0486843Z * [new branch] gh/cccclai/92/head -> origin/gh/cccclai/92/head 2022-03-17T19:07:03.0489647Z * [new branch] gh/cccclai/92/orig -> origin/gh/cccclai/92/orig 2022-03-17T19:07:03.0492009Z * [new branch] gh/cccclai/93/base -> origin/gh/cccclai/93/base 2022-03-17T19:07:03.0494590Z * [new branch] gh/cccclai/93/head -> origin/gh/cccclai/93/head 2022-03-17T19:07:03.0497167Z * [new branch] gh/cccclai/94/base -> origin/gh/cccclai/94/base 2022-03-17T19:07:03.0499526Z * [new branch] gh/cccclai/94/head -> origin/gh/cccclai/94/head 2022-03-17T19:07:03.0501516Z * [new branch] gh/cccclai/94/orig -> origin/gh/cccclai/94/orig 2022-03-17T19:07:03.0505272Z * [new branch] gh/cccclai/95/base -> origin/gh/cccclai/95/base 2022-03-17T19:07:03.0506754Z * [new branch] gh/cccclai/95/head -> origin/gh/cccclai/95/head 2022-03-17T19:07:03.0509611Z * [new branch] gh/cccclai/95/orig -> origin/gh/cccclai/95/orig 2022-03-17T19:07:03.0513080Z * [new branch] gh/cccclai/99/base -> origin/gh/cccclai/99/base 2022-03-17T19:07:03.0514622Z * [new branch] gh/cccclai/99/head -> origin/gh/cccclai/99/head 2022-03-17T19:07:03.0517353Z * [new branch] gh/cccclai/99/orig -> origin/gh/cccclai/99/orig 2022-03-17T19:07:03.0520705Z * [new branch] gh/changjishi/2/base -> origin/gh/changjishi/2/base 2022-03-17T19:07:03.0523421Z * [new branch] gh/changjishi/2/head -> origin/gh/changjishi/2/head 2022-03-17T19:07:03.0525788Z * [new branch] gh/changjishi/2/orig -> origin/gh/changjishi/2/orig 2022-03-17T19:07:03.0529167Z * [new branch] gh/chauhang/1/base -> origin/gh/chauhang/1/base 2022-03-17T19:07:03.0531489Z * [new branch] gh/chauhang/2/base -> origin/gh/chauhang/2/base 2022-03-17T19:07:03.0534881Z * [new branch] gh/cherie11/1/base -> origin/gh/cherie11/1/base 2022-03-17T19:07:03.0536536Z * [new branch] gh/cherie11/1/head -> origin/gh/cherie11/1/head 2022-03-17T19:07:03.0539098Z * [new branch] gh/cherie11/1/orig -> origin/gh/cherie11/1/orig 2022-03-17T19:07:03.0542659Z * [new branch] gh/chillee/42/base -> origin/gh/chillee/42/base 2022-03-17T19:07:03.0545634Z * [new branch] gh/chillee/42/head -> origin/gh/chillee/42/head 2022-03-17T19:07:03.0547461Z * [new branch] gh/chillee/42/orig -> origin/gh/chillee/42/orig 2022-03-17T19:07:03.0550716Z * [new branch] gh/chillee/47/base -> origin/gh/chillee/47/base 2022-03-17T19:07:03.0552274Z * [new branch] gh/chillee/47/head -> origin/gh/chillee/47/head 2022-03-17T19:07:03.0554970Z * [new branch] gh/chillee/47/orig -> origin/gh/chillee/47/orig 2022-03-17T19:07:03.0557662Z * [new branch] gh/chillee/49/base -> origin/gh/chillee/49/base 2022-03-17T19:07:03.0559952Z * [new branch] gh/chillee/49/head -> origin/gh/chillee/49/head 2022-03-17T19:07:03.0561862Z * [new branch] gh/chillee/49/orig -> origin/gh/chillee/49/orig 2022-03-17T19:07:03.0565712Z * [new branch] gh/choward232/4/base -> origin/gh/choward232/4/base 2022-03-17T19:07:03.0567209Z * [new branch] gh/choward232/4/head -> origin/gh/choward232/4/head 2022-03-17T19:07:03.0570196Z * [new branch] gh/choward232/4/orig -> origin/gh/choward232/4/orig 2022-03-17T19:07:03.0573145Z * [new branch] gh/choward232/6/base -> origin/gh/choward232/6/base 2022-03-17T19:07:03.0574863Z * [new branch] gh/choward232/6/head -> origin/gh/choward232/6/head 2022-03-17T19:07:03.0577633Z * [new branch] gh/choward232/6/orig -> origin/gh/choward232/6/orig 2022-03-17T19:07:03.0581294Z * [new branch] gh/dagitses/100/base -> origin/gh/dagitses/100/base 2022-03-17T19:07:03.0583018Z * [new branch] gh/dagitses/100/head -> origin/gh/dagitses/100/head 2022-03-17T19:07:03.0586462Z * [new branch] gh/dagitses/101/base -> origin/gh/dagitses/101/base 2022-03-17T19:07:03.0587882Z * [new branch] gh/dagitses/101/head -> origin/gh/dagitses/101/head 2022-03-17T19:07:03.0590889Z * [new branch] gh/dagitses/102/base -> origin/gh/dagitses/102/base 2022-03-17T19:07:03.0592632Z * [new branch] gh/dagitses/102/head -> origin/gh/dagitses/102/head 2022-03-17T19:07:03.0595930Z * [new branch] gh/dagitses/103/base -> origin/gh/dagitses/103/base 2022-03-17T19:07:03.0597449Z * [new branch] gh/dagitses/103/head -> origin/gh/dagitses/103/head 2022-03-17T19:07:03.0600424Z * [new branch] gh/dagitses/104/base -> origin/gh/dagitses/104/base 2022-03-17T19:07:03.0602199Z * [new branch] gh/dagitses/104/head -> origin/gh/dagitses/104/head 2022-03-17T19:07:03.0605238Z * [new branch] gh/dagitses/105/base -> origin/gh/dagitses/105/base 2022-03-17T19:07:03.0607120Z * [new branch] gh/dagitses/105/head -> origin/gh/dagitses/105/head 2022-03-17T19:07:03.0609392Z * [new branch] gh/dagitses/106/base -> origin/gh/dagitses/106/base 2022-03-17T19:07:03.0611920Z * [new branch] gh/dagitses/106/head -> origin/gh/dagitses/106/head 2022-03-17T19:07:03.0613862Z * [new branch] gh/dagitses/107/base -> origin/gh/dagitses/107/base 2022-03-17T19:07:03.0616128Z * [new branch] gh/dagitses/107/head -> origin/gh/dagitses/107/head 2022-03-17T19:07:03.0618268Z * [new branch] gh/dagitses/108/base -> origin/gh/dagitses/108/base 2022-03-17T19:07:03.0620268Z * [new branch] gh/dagitses/108/head -> origin/gh/dagitses/108/head 2022-03-17T19:07:03.0622907Z * [new branch] gh/dagitses/109/base -> origin/gh/dagitses/109/base 2022-03-17T19:07:03.0624474Z * [new branch] gh/dagitses/109/head -> origin/gh/dagitses/109/head 2022-03-17T19:07:03.0627344Z * [new branch] gh/dagitses/11/base -> origin/gh/dagitses/11/base 2022-03-17T19:07:03.0628883Z * [new branch] gh/dagitses/11/orig -> origin/gh/dagitses/11/orig 2022-03-17T19:07:03.0632187Z * [new branch] gh/dagitses/110/base -> origin/gh/dagitses/110/base 2022-03-17T19:07:03.0633677Z * [new branch] gh/dagitses/110/head -> origin/gh/dagitses/110/head 2022-03-17T19:07:03.0636363Z * [new branch] gh/dagitses/111/base -> origin/gh/dagitses/111/base 2022-03-17T19:07:03.0637917Z * [new branch] gh/dagitses/111/head -> origin/gh/dagitses/111/head 2022-03-17T19:07:03.0640841Z * [new branch] gh/dagitses/112/base -> origin/gh/dagitses/112/base 2022-03-17T19:07:03.0653026Z * [new branch] gh/dagitses/112/head -> origin/gh/dagitses/112/head 2022-03-17T19:07:03.0653930Z * [new branch] gh/dagitses/113/base -> origin/gh/dagitses/113/base 2022-03-17T19:07:03.0654344Z * [new branch] gh/dagitses/113/head -> origin/gh/dagitses/113/head 2022-03-17T19:07:03.0654850Z * [new branch] gh/dagitses/114/base -> origin/gh/dagitses/114/base 2022-03-17T19:07:03.0655439Z * [new branch] gh/dagitses/114/head -> origin/gh/dagitses/114/head 2022-03-17T19:07:03.0656400Z * [new branch] gh/dagitses/115/base -> origin/gh/dagitses/115/base 2022-03-17T19:07:03.0656915Z * [new branch] gh/dagitses/115/head -> origin/gh/dagitses/115/head 2022-03-17T19:07:03.0657552Z * [new branch] gh/dagitses/116/base -> origin/gh/dagitses/116/base 2022-03-17T19:07:03.0658075Z * [new branch] gh/dagitses/116/head -> origin/gh/dagitses/116/head 2022-03-17T19:07:03.0658565Z * [new branch] gh/dagitses/117/base -> origin/gh/dagitses/117/base 2022-03-17T19:07:03.0659012Z * [new branch] gh/dagitses/117/head -> origin/gh/dagitses/117/head 2022-03-17T19:07:03.0660622Z * [new branch] gh/dagitses/118/base -> origin/gh/dagitses/118/base 2022-03-17T19:07:03.0662797Z * [new branch] gh/dagitses/118/head -> origin/gh/dagitses/118/head 2022-03-17T19:07:03.0665612Z * [new branch] gh/dagitses/119/base -> origin/gh/dagitses/119/base 2022-03-17T19:07:03.0667138Z * [new branch] gh/dagitses/119/head -> origin/gh/dagitses/119/head 2022-03-17T19:07:03.0670436Z * [new branch] gh/dagitses/12/base -> origin/gh/dagitses/12/base 2022-03-17T19:07:03.0671991Z * [new branch] gh/dagitses/12/orig -> origin/gh/dagitses/12/orig 2022-03-17T19:07:03.0674749Z * [new branch] gh/dagitses/120/base -> origin/gh/dagitses/120/base 2022-03-17T19:07:03.0676557Z * [new branch] gh/dagitses/120/head -> origin/gh/dagitses/120/head 2022-03-17T19:07:03.0679500Z * [new branch] gh/dagitses/121/base -> origin/gh/dagitses/121/base 2022-03-17T19:07:03.0681122Z * [new branch] gh/dagitses/121/head -> origin/gh/dagitses/121/head 2022-03-17T19:07:03.0685082Z * [new branch] gh/dagitses/122/base -> origin/gh/dagitses/122/base 2022-03-17T19:07:03.0686061Z * [new branch] gh/dagitses/122/head -> origin/gh/dagitses/122/head 2022-03-17T19:07:03.0687867Z * [new branch] gh/dagitses/123/base -> origin/gh/dagitses/123/base 2022-03-17T19:07:03.0689821Z * [new branch] gh/dagitses/123/head -> origin/gh/dagitses/123/head 2022-03-17T19:07:03.0692968Z * [new branch] gh/dagitses/124/base -> origin/gh/dagitses/124/base 2022-03-17T19:07:03.0694487Z * [new branch] gh/dagitses/124/head -> origin/gh/dagitses/124/head 2022-03-17T19:07:03.0698417Z * [new branch] gh/dagitses/125/base -> origin/gh/dagitses/125/base 2022-03-17T19:07:03.0699789Z * [new branch] gh/dagitses/125/head -> origin/gh/dagitses/125/head 2022-03-17T19:07:03.0701960Z * [new branch] gh/dagitses/125/orig -> origin/gh/dagitses/125/orig 2022-03-17T19:07:03.0704963Z * [new branch] gh/dagitses/126/base -> origin/gh/dagitses/126/base 2022-03-17T19:07:03.0706676Z * [new branch] gh/dagitses/126/head -> origin/gh/dagitses/126/head 2022-03-17T19:07:03.0709454Z * [new branch] gh/dagitses/126/orig -> origin/gh/dagitses/126/orig 2022-03-17T19:07:03.0712132Z * [new branch] gh/dagitses/127/base -> origin/gh/dagitses/127/base 2022-03-17T19:07:03.0713939Z * [new branch] gh/dagitses/127/head -> origin/gh/dagitses/127/head 2022-03-17T19:07:03.0716234Z * [new branch] gh/dagitses/127/orig -> origin/gh/dagitses/127/orig 2022-03-17T19:07:03.0720086Z * [new branch] gh/dagitses/128/base -> origin/gh/dagitses/128/base 2022-03-17T19:07:03.0721432Z * [new branch] gh/dagitses/128/head -> origin/gh/dagitses/128/head 2022-03-17T19:07:03.0723821Z * [new branch] gh/dagitses/129/base -> origin/gh/dagitses/129/base 2022-03-17T19:07:03.0725210Z * [new branch] gh/dagitses/129/head -> origin/gh/dagitses/129/head 2022-03-17T19:07:03.0728216Z * [new branch] gh/dagitses/13/base -> origin/gh/dagitses/13/base 2022-03-17T19:07:03.0730510Z * [new branch] gh/dagitses/13/orig -> origin/gh/dagitses/13/orig 2022-03-17T19:07:03.0732688Z * [new branch] gh/dagitses/130/base -> origin/gh/dagitses/130/base 2022-03-17T19:07:03.0734195Z * [new branch] gh/dagitses/130/head -> origin/gh/dagitses/130/head 2022-03-17T19:07:03.0737434Z * [new branch] gh/dagitses/131/base -> origin/gh/dagitses/131/base 2022-03-17T19:07:03.0738978Z * [new branch] gh/dagitses/131/head -> origin/gh/dagitses/131/head 2022-03-17T19:07:03.0741843Z * [new branch] gh/dagitses/132/base -> origin/gh/dagitses/132/base 2022-03-17T19:07:03.0743857Z * [new branch] gh/dagitses/132/head -> origin/gh/dagitses/132/head 2022-03-17T19:07:03.0746074Z * [new branch] gh/dagitses/132/orig -> origin/gh/dagitses/132/orig 2022-03-17T19:07:03.0748892Z * [new branch] gh/dagitses/133/base -> origin/gh/dagitses/133/base 2022-03-17T19:07:03.0750892Z * [new branch] gh/dagitses/133/head -> origin/gh/dagitses/133/head 2022-03-17T19:07:03.0753349Z * [new branch] gh/dagitses/133/orig -> origin/gh/dagitses/133/orig 2022-03-17T19:07:03.0755853Z * [new branch] gh/dagitses/134/base -> origin/gh/dagitses/134/base 2022-03-17T19:07:03.0757892Z * [new branch] gh/dagitses/134/head -> origin/gh/dagitses/134/head 2022-03-17T19:07:03.0760052Z * [new branch] gh/dagitses/134/orig -> origin/gh/dagitses/134/orig 2022-03-17T19:07:03.0762587Z * [new branch] gh/dagitses/135/base -> origin/gh/dagitses/135/base 2022-03-17T19:07:03.0763914Z * [new branch] gh/dagitses/135/head -> origin/gh/dagitses/135/head 2022-03-17T19:07:03.0766443Z * [new branch] gh/dagitses/135/orig -> origin/gh/dagitses/135/orig 2022-03-17T19:07:03.0769002Z * [new branch] gh/dagitses/136/base -> origin/gh/dagitses/136/base 2022-03-17T19:07:03.0770123Z * [new branch] gh/dagitses/136/head -> origin/gh/dagitses/136/head 2022-03-17T19:07:03.0772650Z * [new branch] gh/dagitses/136/orig -> origin/gh/dagitses/136/orig 2022-03-17T19:07:03.0775074Z * [new branch] gh/dagitses/137/base -> origin/gh/dagitses/137/base 2022-03-17T19:07:03.0776425Z * [new branch] gh/dagitses/137/head -> origin/gh/dagitses/137/head 2022-03-17T19:07:03.0779089Z * [new branch] gh/dagitses/137/orig -> origin/gh/dagitses/137/orig 2022-03-17T19:07:03.0781430Z * [new branch] gh/dagitses/138/base -> origin/gh/dagitses/138/base 2022-03-17T19:07:03.0784663Z * [new branch] gh/dagitses/138/head -> origin/gh/dagitses/138/head 2022-03-17T19:07:03.0786420Z * [new branch] gh/dagitses/138/orig -> origin/gh/dagitses/138/orig 2022-03-17T19:07:03.0789062Z * [new branch] gh/dagitses/139/base -> origin/gh/dagitses/139/base 2022-03-17T19:07:03.0791013Z * [new branch] gh/dagitses/139/head -> origin/gh/dagitses/139/head 2022-03-17T19:07:03.0793220Z * [new branch] gh/dagitses/139/orig -> origin/gh/dagitses/139/orig 2022-03-17T19:07:03.0796084Z * [new branch] gh/dagitses/14/base -> origin/gh/dagitses/14/base 2022-03-17T19:07:03.0798073Z * [new branch] gh/dagitses/14/orig -> origin/gh/dagitses/14/orig 2022-03-17T19:07:03.0800871Z * [new branch] gh/dagitses/140/base -> origin/gh/dagitses/140/base 2022-03-17T19:07:03.0802304Z * [new branch] gh/dagitses/140/head -> origin/gh/dagitses/140/head 2022-03-17T19:07:03.0804706Z * [new branch] gh/dagitses/140/orig -> origin/gh/dagitses/140/orig 2022-03-17T19:07:03.0807064Z * [new branch] gh/dagitses/141/base -> origin/gh/dagitses/141/base 2022-03-17T19:07:03.0809161Z * [new branch] gh/dagitses/141/head -> origin/gh/dagitses/141/head 2022-03-17T19:07:03.0810751Z * [new branch] gh/dagitses/141/orig -> origin/gh/dagitses/141/orig 2022-03-17T19:07:03.0813728Z * [new branch] gh/dagitses/142/base -> origin/gh/dagitses/142/base 2022-03-17T19:07:03.0815372Z * [new branch] gh/dagitses/142/head -> origin/gh/dagitses/142/head 2022-03-17T19:07:03.0817899Z * [new branch] gh/dagitses/142/orig -> origin/gh/dagitses/142/orig 2022-03-17T19:07:03.0820293Z * [new branch] gh/dagitses/143/base -> origin/gh/dagitses/143/base 2022-03-17T19:07:03.0821793Z * [new branch] gh/dagitses/143/head -> origin/gh/dagitses/143/head 2022-03-17T19:07:03.0824364Z * [new branch] gh/dagitses/143/orig -> origin/gh/dagitses/143/orig 2022-03-17T19:07:03.0826754Z * [new branch] gh/dagitses/144/base -> origin/gh/dagitses/144/base 2022-03-17T19:07:03.0828684Z * [new branch] gh/dagitses/144/head -> origin/gh/dagitses/144/head 2022-03-17T19:07:03.0830760Z * [new branch] gh/dagitses/144/orig -> origin/gh/dagitses/144/orig 2022-03-17T19:07:03.0833313Z * [new branch] gh/dagitses/145/base -> origin/gh/dagitses/145/base 2022-03-17T19:07:03.0834905Z * [new branch] gh/dagitses/145/head -> origin/gh/dagitses/145/head 2022-03-17T19:07:03.0837072Z * [new branch] gh/dagitses/145/orig -> origin/gh/dagitses/145/orig 2022-03-17T19:07:03.0839903Z * [new branch] gh/dagitses/146/base -> origin/gh/dagitses/146/base 2022-03-17T19:07:03.0841080Z * [new branch] gh/dagitses/146/head -> origin/gh/dagitses/146/head 2022-03-17T19:07:03.0843911Z * [new branch] gh/dagitses/146/orig -> origin/gh/dagitses/146/orig 2022-03-17T19:07:03.0844962Z * [new branch] gh/dagitses/147/base -> origin/gh/dagitses/147/base 2022-03-17T19:07:03.0847851Z * [new branch] gh/dagitses/147/head -> origin/gh/dagitses/147/head 2022-03-17T19:07:03.0849044Z * [new branch] gh/dagitses/148/base -> origin/gh/dagitses/148/base 2022-03-17T19:07:03.0850532Z * [new branch] gh/dagitses/148/head -> origin/gh/dagitses/148/head 2022-03-17T19:07:03.0852096Z * [new branch] gh/dagitses/149/base -> origin/gh/dagitses/149/base 2022-03-17T19:07:03.0853584Z * [new branch] gh/dagitses/149/head -> origin/gh/dagitses/149/head 2022-03-17T19:07:03.0855808Z * [new branch] gh/dagitses/15/base -> origin/gh/dagitses/15/base 2022-03-17T19:07:03.0856576Z * [new branch] gh/dagitses/15/orig -> origin/gh/dagitses/15/orig 2022-03-17T19:07:03.0858669Z * [new branch] gh/dagitses/150/base -> origin/gh/dagitses/150/base 2022-03-17T19:07:03.0859750Z * [new branch] gh/dagitses/150/head -> origin/gh/dagitses/150/head 2022-03-17T19:07:03.0861437Z * [new branch] gh/dagitses/151/base -> origin/gh/dagitses/151/base 2022-03-17T19:07:03.0862896Z * [new branch] gh/dagitses/151/head -> origin/gh/dagitses/151/head 2022-03-17T19:07:03.0864450Z * [new branch] gh/dagitses/152/base -> origin/gh/dagitses/152/base 2022-03-17T19:07:03.0865827Z * [new branch] gh/dagitses/152/head -> origin/gh/dagitses/152/head 2022-03-17T19:07:03.0867616Z * [new branch] gh/dagitses/153/base -> origin/gh/dagitses/153/base 2022-03-17T19:07:03.0868459Z * [new branch] gh/dagitses/153/head -> origin/gh/dagitses/153/head 2022-03-17T19:07:03.0870340Z * [new branch] gh/dagitses/154/base -> origin/gh/dagitses/154/base 2022-03-17T19:07:03.0871565Z * [new branch] gh/dagitses/154/head -> origin/gh/dagitses/154/head 2022-03-17T19:07:03.0873159Z * [new branch] gh/dagitses/155/base -> origin/gh/dagitses/155/base 2022-03-17T19:07:03.0875274Z * [new branch] gh/dagitses/155/head -> origin/gh/dagitses/155/head 2022-03-17T19:07:03.0876543Z * [new branch] gh/dagitses/156/base -> origin/gh/dagitses/156/base 2022-03-17T19:07:03.0878098Z * [new branch] gh/dagitses/156/head -> origin/gh/dagitses/156/head 2022-03-17T19:07:03.0879580Z * [new branch] gh/dagitses/157/base -> origin/gh/dagitses/157/base 2022-03-17T19:07:03.0880884Z * [new branch] gh/dagitses/157/head -> origin/gh/dagitses/157/head 2022-03-17T19:07:03.0882679Z * [new branch] gh/dagitses/158/base -> origin/gh/dagitses/158/base 2022-03-17T19:07:03.0884687Z * [new branch] gh/dagitses/158/head -> origin/gh/dagitses/158/head 2022-03-17T19:07:03.0885571Z * [new branch] gh/dagitses/159/base -> origin/gh/dagitses/159/base 2022-03-17T19:07:03.0886875Z * [new branch] gh/dagitses/159/head -> origin/gh/dagitses/159/head 2022-03-17T19:07:03.0888952Z * [new branch] gh/dagitses/16/base -> origin/gh/dagitses/16/base 2022-03-17T19:07:03.0890241Z * [new branch] gh/dagitses/16/orig -> origin/gh/dagitses/16/orig 2022-03-17T19:07:03.0891595Z * [new branch] gh/dagitses/160/base -> origin/gh/dagitses/160/base 2022-03-17T19:07:03.0892984Z * [new branch] gh/dagitses/160/head -> origin/gh/dagitses/160/head 2022-03-17T19:07:03.0894859Z * [new branch] gh/dagitses/161/base -> origin/gh/dagitses/161/base 2022-03-17T19:07:03.0895963Z * [new branch] gh/dagitses/161/head -> origin/gh/dagitses/161/head 2022-03-17T19:07:03.0897608Z * [new branch] gh/dagitses/162/base -> origin/gh/dagitses/162/base 2022-03-17T19:07:03.0898877Z * [new branch] gh/dagitses/162/head -> origin/gh/dagitses/162/head 2022-03-17T19:07:03.0900892Z * [new branch] gh/dagitses/163/base -> origin/gh/dagitses/163/base 2022-03-17T19:07:03.0902079Z * [new branch] gh/dagitses/163/head -> origin/gh/dagitses/163/head 2022-03-17T19:07:03.0903596Z * [new branch] gh/dagitses/164/base -> origin/gh/dagitses/164/base 2022-03-17T19:07:03.0904813Z * [new branch] gh/dagitses/164/head -> origin/gh/dagitses/164/head 2022-03-17T19:07:03.0906390Z * [new branch] gh/dagitses/165/base -> origin/gh/dagitses/165/base 2022-03-17T19:07:03.0907732Z * [new branch] gh/dagitses/165/head -> origin/gh/dagitses/165/head 2022-03-17T19:07:03.0909124Z * [new branch] gh/dagitses/166/base -> origin/gh/dagitses/166/base 2022-03-17T19:07:03.0910303Z * [new branch] gh/dagitses/166/head -> origin/gh/dagitses/166/head 2022-03-17T19:07:03.0912696Z * [new branch] gh/dagitses/167/base -> origin/gh/dagitses/167/base 2022-03-17T19:07:03.0913452Z * [new branch] gh/dagitses/167/head -> origin/gh/dagitses/167/head 2022-03-17T19:07:03.0915487Z * [new branch] gh/dagitses/168/base -> origin/gh/dagitses/168/base 2022-03-17T19:07:03.0916568Z * [new branch] gh/dagitses/168/head -> origin/gh/dagitses/168/head 2022-03-17T19:07:03.0918626Z * [new branch] gh/dagitses/169/base -> origin/gh/dagitses/169/base 2022-03-17T19:07:03.0919794Z * [new branch] gh/dagitses/169/head -> origin/gh/dagitses/169/head 2022-03-17T19:07:03.0921948Z * [new branch] gh/dagitses/17/base -> origin/gh/dagitses/17/base 2022-03-17T19:07:03.0922975Z * [new branch] gh/dagitses/17/orig -> origin/gh/dagitses/17/orig 2022-03-17T19:07:03.0925157Z * [new branch] gh/dagitses/170/base -> origin/gh/dagitses/170/base 2022-03-17T19:07:03.0926172Z * [new branch] gh/dagitses/170/head -> origin/gh/dagitses/170/head 2022-03-17T19:07:03.0927859Z * [new branch] gh/dagitses/171/base -> origin/gh/dagitses/171/base 2022-03-17T19:07:03.0929205Z * [new branch] gh/dagitses/171/head -> origin/gh/dagitses/171/head 2022-03-17T19:07:03.0930813Z * [new branch] gh/dagitses/172/base -> origin/gh/dagitses/172/base 2022-03-17T19:07:03.0932139Z * [new branch] gh/dagitses/172/head -> origin/gh/dagitses/172/head 2022-03-17T19:07:03.0933674Z * [new branch] gh/dagitses/173/base -> origin/gh/dagitses/173/base 2022-03-17T19:07:03.0935806Z * [new branch] gh/dagitses/173/head -> origin/gh/dagitses/173/head 2022-03-17T19:07:03.0936822Z * [new branch] gh/dagitses/174/base -> origin/gh/dagitses/174/base 2022-03-17T19:07:03.0938134Z * [new branch] gh/dagitses/174/head -> origin/gh/dagitses/174/head 2022-03-17T19:07:03.0939660Z * [new branch] gh/dagitses/175/base -> origin/gh/dagitses/175/base 2022-03-17T19:07:03.0941120Z * [new branch] gh/dagitses/175/head -> origin/gh/dagitses/175/head 2022-03-17T19:07:03.0942950Z * [new branch] gh/dagitses/176/base -> origin/gh/dagitses/176/base 2022-03-17T19:07:03.0943894Z * [new branch] gh/dagitses/176/head -> origin/gh/dagitses/176/head 2022-03-17T19:07:03.0945966Z * [new branch] gh/dagitses/177/base -> origin/gh/dagitses/177/base 2022-03-17T19:07:03.0947006Z * [new branch] gh/dagitses/177/head -> origin/gh/dagitses/177/head 2022-03-17T19:07:03.0948603Z * [new branch] gh/dagitses/178/base -> origin/gh/dagitses/178/base 2022-03-17T19:07:03.0949935Z * [new branch] gh/dagitses/178/head -> origin/gh/dagitses/178/head 2022-03-17T19:07:03.0952093Z * [new branch] gh/dagitses/179/base -> origin/gh/dagitses/179/base 2022-03-17T19:07:03.0953803Z * [new branch] gh/dagitses/179/head -> origin/gh/dagitses/179/head 2022-03-17T19:07:03.0956163Z * [new branch] gh/dagitses/18/base -> origin/gh/dagitses/18/base 2022-03-17T19:07:03.0958052Z * [new branch] gh/dagitses/18/orig -> origin/gh/dagitses/18/orig 2022-03-17T19:07:03.0960271Z * [new branch] gh/dagitses/180/base -> origin/gh/dagitses/180/base 2022-03-17T19:07:03.0961153Z * [new branch] gh/dagitses/180/head -> origin/gh/dagitses/180/head 2022-03-17T19:07:03.0963305Z * [new branch] gh/dagitses/181/base -> origin/gh/dagitses/181/base 2022-03-17T19:07:03.0964541Z * [new branch] gh/dagitses/181/head -> origin/gh/dagitses/181/head 2022-03-17T19:07:03.0965977Z * [new branch] gh/dagitses/182/base -> origin/gh/dagitses/182/base 2022-03-17T19:07:03.0967164Z * [new branch] gh/dagitses/182/head -> origin/gh/dagitses/182/head 2022-03-17T19:07:03.0968663Z * [new branch] gh/dagitses/183/base -> origin/gh/dagitses/183/base 2022-03-17T19:07:03.0970112Z * [new branch] gh/dagitses/183/head -> origin/gh/dagitses/183/head 2022-03-17T19:07:03.0972790Z * [new branch] gh/dagitses/184/base -> origin/gh/dagitses/184/base 2022-03-17T19:07:03.0973633Z * [new branch] gh/dagitses/184/head -> origin/gh/dagitses/184/head 2022-03-17T19:07:03.0975318Z * [new branch] gh/dagitses/185/base -> origin/gh/dagitses/185/base 2022-03-17T19:07:03.0976609Z * [new branch] gh/dagitses/185/head -> origin/gh/dagitses/185/head 2022-03-17T19:07:03.0978789Z * [new branch] gh/dagitses/186/base -> origin/gh/dagitses/186/base 2022-03-17T19:07:03.0979876Z * [new branch] gh/dagitses/186/head -> origin/gh/dagitses/186/head 2022-03-17T19:07:03.0982686Z * [new branch] gh/dagitses/187/base -> origin/gh/dagitses/187/base 2022-03-17T19:07:03.0983714Z * [new branch] gh/dagitses/187/head -> origin/gh/dagitses/187/head 2022-03-17T19:07:03.0985216Z * [new branch] gh/dagitses/188/base -> origin/gh/dagitses/188/base 2022-03-17T19:07:03.0986429Z * [new branch] gh/dagitses/188/head -> origin/gh/dagitses/188/head 2022-03-17T19:07:03.0987974Z * [new branch] gh/dagitses/189/base -> origin/gh/dagitses/189/base 2022-03-17T19:07:03.0989383Z * [new branch] gh/dagitses/189/head -> origin/gh/dagitses/189/head 2022-03-17T19:07:03.0991471Z * [new branch] gh/dagitses/19/base -> origin/gh/dagitses/19/base 2022-03-17T19:07:03.0992721Z * [new branch] gh/dagitses/19/orig -> origin/gh/dagitses/19/orig 2022-03-17T19:07:03.0994208Z * [new branch] gh/dagitses/190/base -> origin/gh/dagitses/190/base 2022-03-17T19:07:03.0995425Z * [new branch] gh/dagitses/190/head -> origin/gh/dagitses/190/head 2022-03-17T19:07:03.0996973Z * [new branch] gh/dagitses/191/base -> origin/gh/dagitses/191/base 2022-03-17T19:07:03.0998219Z * [new branch] gh/dagitses/191/head -> origin/gh/dagitses/191/head 2022-03-17T19:07:03.0999713Z * [new branch] gh/dagitses/192/base -> origin/gh/dagitses/192/base 2022-03-17T19:07:03.1001123Z * [new branch] gh/dagitses/192/head -> origin/gh/dagitses/192/head 2022-03-17T19:07:03.1002672Z * [new branch] gh/dagitses/193/base -> origin/gh/dagitses/193/base 2022-03-17T19:07:03.1003913Z * [new branch] gh/dagitses/193/head -> origin/gh/dagitses/193/head 2022-03-17T19:07:03.1005771Z * [new branch] gh/dagitses/194/base -> origin/gh/dagitses/194/base 2022-03-17T19:07:03.1006915Z * [new branch] gh/dagitses/194/head -> origin/gh/dagitses/194/head 2022-03-17T19:07:03.1008591Z * [new branch] gh/dagitses/195/base -> origin/gh/dagitses/195/base 2022-03-17T19:07:03.1010774Z * [new branch] gh/dagitses/195/head -> origin/gh/dagitses/195/head 2022-03-17T19:07:03.1012044Z * [new branch] gh/dagitses/196/base -> origin/gh/dagitses/196/base 2022-03-17T19:07:03.1013336Z * [new branch] gh/dagitses/196/head -> origin/gh/dagitses/196/head 2022-03-17T19:07:03.1015800Z * [new branch] gh/dagitses/197/base -> origin/gh/dagitses/197/base 2022-03-17T19:07:03.1016921Z * [new branch] gh/dagitses/197/head -> origin/gh/dagitses/197/head 2022-03-17T19:07:03.1018528Z * [new branch] gh/dagitses/198/base -> origin/gh/dagitses/198/base 2022-03-17T19:07:03.1019814Z * [new branch] gh/dagitses/198/head -> origin/gh/dagitses/198/head 2022-03-17T19:07:03.1021422Z * [new branch] gh/dagitses/199/base -> origin/gh/dagitses/199/base 2022-03-17T19:07:03.1022910Z * [new branch] gh/dagitses/199/head -> origin/gh/dagitses/199/head 2022-03-17T19:07:03.1025392Z * [new branch] gh/dagitses/20/base -> origin/gh/dagitses/20/base 2022-03-17T19:07:03.1026326Z * [new branch] gh/dagitses/20/orig -> origin/gh/dagitses/20/orig 2022-03-17T19:07:03.1028255Z * [new branch] gh/dagitses/200/base -> origin/gh/dagitses/200/base 2022-03-17T19:07:03.1029006Z * [new branch] gh/dagitses/200/head -> origin/gh/dagitses/200/head 2022-03-17T19:07:03.1030982Z * [new branch] gh/dagitses/201/base -> origin/gh/dagitses/201/base 2022-03-17T19:07:03.1032253Z * [new branch] gh/dagitses/201/head -> origin/gh/dagitses/201/head 2022-03-17T19:07:03.1034258Z * [new branch] gh/dagitses/202/base -> origin/gh/dagitses/202/base 2022-03-17T19:07:03.1035372Z * [new branch] gh/dagitses/202/head -> origin/gh/dagitses/202/head 2022-03-17T19:07:03.1036909Z * [new branch] gh/dagitses/203/base -> origin/gh/dagitses/203/base 2022-03-17T19:07:03.1038276Z * [new branch] gh/dagitses/203/head -> origin/gh/dagitses/203/head 2022-03-17T19:07:03.1039874Z * [new branch] gh/dagitses/204/base -> origin/gh/dagitses/204/base 2022-03-17T19:07:03.1041430Z * [new branch] gh/dagitses/204/head -> origin/gh/dagitses/204/head 2022-03-17T19:07:03.1042962Z * [new branch] gh/dagitses/205/base -> origin/gh/dagitses/205/base 2022-03-17T19:07:03.1044358Z * [new branch] gh/dagitses/205/head -> origin/gh/dagitses/205/head 2022-03-17T19:07:03.1045890Z * [new branch] gh/dagitses/206/base -> origin/gh/dagitses/206/base 2022-03-17T19:07:03.1047342Z * [new branch] gh/dagitses/206/head -> origin/gh/dagitses/206/head 2022-03-17T19:07:03.1048919Z * [new branch] gh/dagitses/207/base -> origin/gh/dagitses/207/base 2022-03-17T19:07:03.1051087Z * [new branch] gh/dagitses/207/head -> origin/gh/dagitses/207/head 2022-03-17T19:07:03.1052395Z * [new branch] gh/dagitses/208/base -> origin/gh/dagitses/208/base 2022-03-17T19:07:03.1053859Z * [new branch] gh/dagitses/208/head -> origin/gh/dagitses/208/head 2022-03-17T19:07:03.1055376Z * [new branch] gh/dagitses/209/base -> origin/gh/dagitses/209/base 2022-03-17T19:07:03.1057570Z * [new branch] gh/dagitses/209/head -> origin/gh/dagitses/209/head 2022-03-17T19:07:03.1059273Z * [new branch] gh/dagitses/21/base -> origin/gh/dagitses/21/base 2022-03-17T19:07:03.1061097Z * [new branch] gh/dagitses/21/orig -> origin/gh/dagitses/21/orig 2022-03-17T19:07:03.1063226Z * [new branch] gh/dagitses/210/base -> origin/gh/dagitses/210/base 2022-03-17T19:07:03.1064198Z * [new branch] gh/dagitses/210/head -> origin/gh/dagitses/210/head 2022-03-17T19:07:03.1065754Z * [new branch] gh/dagitses/211/base -> origin/gh/dagitses/211/base 2022-03-17T19:07:03.1067212Z * [new branch] gh/dagitses/211/head -> origin/gh/dagitses/211/head 2022-03-17T19:07:03.1068765Z * [new branch] gh/dagitses/212/base -> origin/gh/dagitses/212/base 2022-03-17T19:07:03.1070889Z * [new branch] gh/dagitses/212/head -> origin/gh/dagitses/212/head 2022-03-17T19:07:03.1072131Z * [new branch] gh/dagitses/213/base -> origin/gh/dagitses/213/base 2022-03-17T19:07:03.1073623Z * [new branch] gh/dagitses/213/head -> origin/gh/dagitses/213/head 2022-03-17T19:07:03.1075362Z * [new branch] gh/dagitses/214/base -> origin/gh/dagitses/214/base 2022-03-17T19:07:03.1076594Z * [new branch] gh/dagitses/214/head -> origin/gh/dagitses/214/head 2022-03-17T19:07:03.1078253Z * [new branch] gh/dagitses/215/base -> origin/gh/dagitses/215/base 2022-03-17T19:07:03.1080602Z * [new branch] gh/dagitses/215/head -> origin/gh/dagitses/215/head 2022-03-17T19:07:03.1082612Z * [new branch] gh/dagitses/216/base -> origin/gh/dagitses/216/base 2022-03-17T19:07:03.1084766Z * [new branch] gh/dagitses/216/head -> origin/gh/dagitses/216/head 2022-03-17T19:07:03.1086362Z * [new branch] gh/dagitses/217/base -> origin/gh/dagitses/217/base 2022-03-17T19:07:03.1087663Z * [new branch] gh/dagitses/217/head -> origin/gh/dagitses/217/head 2022-03-17T19:07:03.1089189Z * [new branch] gh/dagitses/218/base -> origin/gh/dagitses/218/base 2022-03-17T19:07:03.1090409Z * [new branch] gh/dagitses/218/head -> origin/gh/dagitses/218/head 2022-03-17T19:07:03.1092000Z * [new branch] gh/dagitses/219/base -> origin/gh/dagitses/219/base 2022-03-17T19:07:03.1093275Z * [new branch] gh/dagitses/219/head -> origin/gh/dagitses/219/head 2022-03-17T19:07:03.1094797Z * [new branch] gh/dagitses/220/base -> origin/gh/dagitses/220/base 2022-03-17T19:07:03.1096245Z * [new branch] gh/dagitses/220/head -> origin/gh/dagitses/220/head 2022-03-17T19:07:03.1097766Z * [new branch] gh/dagitses/221/base -> origin/gh/dagitses/221/base 2022-03-17T19:07:03.1099032Z * [new branch] gh/dagitses/221/head -> origin/gh/dagitses/221/head 2022-03-17T19:07:03.1100762Z * [new branch] gh/dagitses/222/base -> origin/gh/dagitses/222/base 2022-03-17T19:07:03.1102104Z * [new branch] gh/dagitses/222/head -> origin/gh/dagitses/222/head 2022-03-17T19:07:03.1103588Z * [new branch] gh/dagitses/223/base -> origin/gh/dagitses/223/base 2022-03-17T19:07:03.1105667Z * [new branch] gh/dagitses/223/head -> origin/gh/dagitses/223/head 2022-03-17T19:07:03.1106868Z * [new branch] gh/dagitses/224/base -> origin/gh/dagitses/224/base 2022-03-17T19:07:03.1108845Z * [new branch] gh/dagitses/224/head -> origin/gh/dagitses/224/head 2022-03-17T19:07:03.1109919Z * [new branch] gh/dagitses/224/orig -> origin/gh/dagitses/224/orig 2022-03-17T19:07:03.1111426Z * [new branch] gh/dagitses/225/base -> origin/gh/dagitses/225/base 2022-03-17T19:07:03.1112817Z * [new branch] gh/dagitses/225/head -> origin/gh/dagitses/225/head 2022-03-17T19:07:03.1114385Z * [new branch] gh/dagitses/225/orig -> origin/gh/dagitses/225/orig 2022-03-17T19:07:03.1116495Z * [new branch] gh/dagitses/226/base -> origin/gh/dagitses/226/base 2022-03-17T19:07:03.1117926Z * [new branch] gh/dagitses/226/head -> origin/gh/dagitses/226/head 2022-03-17T19:07:03.1119419Z * [new branch] gh/dagitses/226/orig -> origin/gh/dagitses/226/orig 2022-03-17T19:07:03.1120883Z * [new branch] gh/dagitses/227/base -> origin/gh/dagitses/227/base 2022-03-17T19:07:03.1122191Z * [new branch] gh/dagitses/227/head -> origin/gh/dagitses/227/head 2022-03-17T19:07:03.1123462Z * [new branch] gh/dagitses/227/orig -> origin/gh/dagitses/227/orig 2022-03-17T19:07:03.1125053Z * [new branch] gh/dagitses/228/base -> origin/gh/dagitses/228/base 2022-03-17T19:07:03.1126279Z * [new branch] gh/dagitses/228/head -> origin/gh/dagitses/228/head 2022-03-17T19:07:03.1127646Z * [new branch] gh/dagitses/228/orig -> origin/gh/dagitses/228/orig 2022-03-17T19:07:03.1129043Z * [new branch] gh/dagitses/229/base -> origin/gh/dagitses/229/base 2022-03-17T19:07:03.1130616Z * [new branch] gh/dagitses/229/head -> origin/gh/dagitses/229/head 2022-03-17T19:07:03.1131901Z * [new branch] gh/dagitses/229/orig -> origin/gh/dagitses/229/orig 2022-03-17T19:07:03.1134253Z * [new branch] gh/dagitses/23/base -> origin/gh/dagitses/23/base 2022-03-17T19:07:03.1135325Z * [new branch] gh/dagitses/23/orig -> origin/gh/dagitses/23/orig 2022-03-17T19:07:03.1136853Z * [new branch] gh/dagitses/230/base -> origin/gh/dagitses/230/base 2022-03-17T19:07:03.1138061Z * [new branch] gh/dagitses/230/head -> origin/gh/dagitses/230/head 2022-03-17T19:07:03.1139430Z * [new branch] gh/dagitses/230/orig -> origin/gh/dagitses/230/orig 2022-03-17T19:07:03.1141204Z * [new branch] gh/dagitses/231/base -> origin/gh/dagitses/231/base 2022-03-17T19:07:03.1142647Z * [new branch] gh/dagitses/231/head -> origin/gh/dagitses/231/head 2022-03-17T19:07:03.1144163Z * [new branch] gh/dagitses/231/orig -> origin/gh/dagitses/231/orig 2022-03-17T19:07:03.1145620Z * [new branch] gh/dagitses/232/base -> origin/gh/dagitses/232/base 2022-03-17T19:07:03.1146955Z * [new branch] gh/dagitses/232/head -> origin/gh/dagitses/232/head 2022-03-17T19:07:03.1148354Z * [new branch] gh/dagitses/232/orig -> origin/gh/dagitses/232/orig 2022-03-17T19:07:03.1149854Z * [new branch] gh/dagitses/233/base -> origin/gh/dagitses/233/base 2022-03-17T19:07:03.1151438Z * [new branch] gh/dagitses/233/head -> origin/gh/dagitses/233/head 2022-03-17T19:07:03.1152882Z * [new branch] gh/dagitses/233/orig -> origin/gh/dagitses/233/orig 2022-03-17T19:07:03.1154348Z * [new branch] gh/dagitses/234/base -> origin/gh/dagitses/234/base 2022-03-17T19:07:03.1155596Z * [new branch] gh/dagitses/234/head -> origin/gh/dagitses/234/head 2022-03-17T19:07:03.1157416Z * [new branch] gh/dagitses/234/orig -> origin/gh/dagitses/234/orig 2022-03-17T19:07:03.1158856Z * [new branch] gh/dagitses/235/base -> origin/gh/dagitses/235/base 2022-03-17T19:07:03.1161103Z * [new branch] gh/dagitses/235/head -> origin/gh/dagitses/235/head 2022-03-17T19:07:03.1162191Z * [new branch] gh/dagitses/235/orig -> origin/gh/dagitses/235/orig 2022-03-17T19:07:03.1163897Z * [new branch] gh/dagitses/236/base -> origin/gh/dagitses/236/base 2022-03-17T19:07:03.1165247Z * [new branch] gh/dagitses/236/head -> origin/gh/dagitses/236/head 2022-03-17T19:07:03.1166675Z * [new branch] gh/dagitses/236/orig -> origin/gh/dagitses/236/orig 2022-03-17T19:07:03.1168120Z * [new branch] gh/dagitses/237/base -> origin/gh/dagitses/237/base 2022-03-17T19:07:03.1169509Z * [new branch] gh/dagitses/237/head -> origin/gh/dagitses/237/head 2022-03-17T19:07:03.1171076Z * [new branch] gh/dagitses/237/orig -> origin/gh/dagitses/237/orig 2022-03-17T19:07:03.1172637Z * [new branch] gh/dagitses/238/base -> origin/gh/dagitses/238/base 2022-03-17T19:07:03.1173931Z * [new branch] gh/dagitses/238/head -> origin/gh/dagitses/238/head 2022-03-17T19:07:03.1175311Z * [new branch] gh/dagitses/238/orig -> origin/gh/dagitses/238/orig 2022-03-17T19:07:03.1177622Z * [new branch] gh/dagitses/239/base -> origin/gh/dagitses/239/base 2022-03-17T19:07:03.1178615Z * [new branch] gh/dagitses/239/head -> origin/gh/dagitses/239/head 2022-03-17T19:07:03.1179846Z * [new branch] gh/dagitses/239/orig -> origin/gh/dagitses/239/orig 2022-03-17T19:07:03.1182733Z * [new branch] gh/dagitses/240/base -> origin/gh/dagitses/240/base 2022-03-17T19:07:03.1184115Z * [new branch] gh/dagitses/240/head -> origin/gh/dagitses/240/head 2022-03-17T19:07:03.1185535Z * [new branch] gh/dagitses/240/orig -> origin/gh/dagitses/240/orig 2022-03-17T19:07:03.1187758Z * [new branch] gh/dagitses/241/base -> origin/gh/dagitses/241/base 2022-03-17T19:07:03.1188925Z * [new branch] gh/dagitses/241/head -> origin/gh/dagitses/241/head 2022-03-17T19:07:03.1191800Z * [new branch] gh/dagitses/241/orig -> origin/gh/dagitses/241/orig 2022-03-17T19:07:03.1193716Z * [new branch] gh/dagitses/244/base -> origin/gh/dagitses/244/base 2022-03-17T19:07:03.1196347Z * [new branch] gh/dagitses/244/head -> origin/gh/dagitses/244/head 2022-03-17T19:07:03.1197677Z * [new branch] gh/dagitses/244/orig -> origin/gh/dagitses/244/orig 2022-03-17T19:07:03.1200934Z * [new branch] gh/dagitses/245/base -> origin/gh/dagitses/245/base 2022-03-17T19:07:03.1203011Z * [new branch] gh/dagitses/245/head -> origin/gh/dagitses/245/head 2022-03-17T19:07:03.1204499Z * [new branch] gh/dagitses/245/orig -> origin/gh/dagitses/245/orig 2022-03-17T19:07:03.1207530Z * [new branch] gh/dagitses/25/base -> origin/gh/dagitses/25/base 2022-03-17T19:07:03.1209296Z * [new branch] gh/dagitses/25/orig -> origin/gh/dagitses/25/orig 2022-03-17T19:07:03.1212387Z * [new branch] gh/dagitses/257/base -> origin/gh/dagitses/257/base 2022-03-17T19:07:03.1213005Z * [new branch] gh/dagitses/257/head -> origin/gh/dagitses/257/head 2022-03-17T19:07:03.1215659Z * [new branch] gh/dagitses/257/orig -> origin/gh/dagitses/257/orig 2022-03-17T19:07:03.1218897Z * [new branch] gh/dagitses/26/base -> origin/gh/dagitses/26/base 2022-03-17T19:07:03.1221079Z * [new branch] gh/dagitses/26/orig -> origin/gh/dagitses/26/orig 2022-03-17T19:07:03.1223922Z * [new branch] gh/dagitses/262/base -> origin/gh/dagitses/262/base 2022-03-17T19:07:03.1226022Z * [new branch] gh/dagitses/262/head -> origin/gh/dagitses/262/head 2022-03-17T19:07:03.1228184Z * [new branch] gh/dagitses/262/orig -> origin/gh/dagitses/262/orig 2022-03-17T19:07:03.1230786Z * [new branch] gh/dagitses/263/base -> origin/gh/dagitses/263/base 2022-03-17T19:07:03.1232272Z * [new branch] gh/dagitses/263/head -> origin/gh/dagitses/263/head 2022-03-17T19:07:03.1235372Z * [new branch] gh/dagitses/263/orig -> origin/gh/dagitses/263/orig 2022-03-17T19:07:03.1237784Z * [new branch] gh/dagitses/264/base -> origin/gh/dagitses/264/base 2022-03-17T19:07:03.1239799Z * [new branch] gh/dagitses/264/head -> origin/gh/dagitses/264/head 2022-03-17T19:07:03.1242005Z * [new branch] gh/dagitses/264/orig -> origin/gh/dagitses/264/orig 2022-03-17T19:07:03.1244705Z * [new branch] gh/dagitses/265/base -> origin/gh/dagitses/265/base 2022-03-17T19:07:03.1246605Z * [new branch] gh/dagitses/265/head -> origin/gh/dagitses/265/head 2022-03-17T19:07:03.1248967Z * [new branch] gh/dagitses/265/orig -> origin/gh/dagitses/265/orig 2022-03-17T19:07:03.1252043Z * [new branch] gh/dagitses/266/base -> origin/gh/dagitses/266/base 2022-03-17T19:07:03.1253920Z * [new branch] gh/dagitses/266/head -> origin/gh/dagitses/266/head 2022-03-17T19:07:03.1256403Z * [new branch] gh/dagitses/266/orig -> origin/gh/dagitses/266/orig 2022-03-17T19:07:03.1258797Z * [new branch] gh/dagitses/271/base -> origin/gh/dagitses/271/base 2022-03-17T19:07:03.1260764Z * [new branch] gh/dagitses/271/head -> origin/gh/dagitses/271/head 2022-03-17T19:07:03.1263178Z * [new branch] gh/dagitses/271/orig -> origin/gh/dagitses/271/orig 2022-03-17T19:07:03.1266110Z * [new branch] gh/dagitses/272/base -> origin/gh/dagitses/272/base 2022-03-17T19:07:03.1267547Z * [new branch] gh/dagitses/272/head -> origin/gh/dagitses/272/head 2022-03-17T19:07:03.1270054Z * [new branch] gh/dagitses/272/orig -> origin/gh/dagitses/272/orig 2022-03-17T19:07:03.1272700Z * [new branch] gh/dagitses/273/base -> origin/gh/dagitses/273/base 2022-03-17T19:07:03.1274733Z * [new branch] gh/dagitses/273/head -> origin/gh/dagitses/273/head 2022-03-17T19:07:03.1276854Z * [new branch] gh/dagitses/273/orig -> origin/gh/dagitses/273/orig 2022-03-17T19:07:03.1279788Z * [new branch] gh/dagitses/275/base -> origin/gh/dagitses/275/base 2022-03-17T19:07:03.1282448Z * [new branch] gh/dagitses/275/head -> origin/gh/dagitses/275/head 2022-03-17T19:07:03.1284134Z * [new branch] gh/dagitses/275/orig -> origin/gh/dagitses/275/orig 2022-03-17T19:07:03.1286792Z * [new branch] gh/dagitses/277/base -> origin/gh/dagitses/277/base 2022-03-17T19:07:03.1288642Z * [new branch] gh/dagitses/277/head -> origin/gh/dagitses/277/head 2022-03-17T19:07:03.1290843Z * [new branch] gh/dagitses/277/orig -> origin/gh/dagitses/277/orig 2022-03-17T19:07:03.1293391Z * [new branch] gh/dagitses/278/base -> origin/gh/dagitses/278/base 2022-03-17T19:07:03.1295421Z * [new branch] gh/dagitses/278/head -> origin/gh/dagitses/278/head 2022-03-17T19:07:03.1297441Z * [new branch] gh/dagitses/278/orig -> origin/gh/dagitses/278/orig 2022-03-17T19:07:03.1300096Z * [new branch] gh/dagitses/279/base -> origin/gh/dagitses/279/base 2022-03-17T19:07:03.1301728Z * [new branch] gh/dagitses/279/orig -> origin/gh/dagitses/279/orig 2022-03-17T19:07:03.1304561Z * [new branch] gh/dagitses/280/base -> origin/gh/dagitses/280/base 2022-03-17T19:07:03.1306411Z * [new branch] gh/dagitses/280/orig -> origin/gh/dagitses/280/orig 2022-03-17T19:07:03.1309187Z * [new branch] gh/dagitses/281/base -> origin/gh/dagitses/281/base 2022-03-17T19:07:03.1311032Z * [new branch] gh/dagitses/281/orig -> origin/gh/dagitses/281/orig 2022-03-17T19:07:03.1313615Z * [new branch] gh/dagitses/282/base -> origin/gh/dagitses/282/base 2022-03-17T19:07:03.1315464Z * [new branch] gh/dagitses/282/orig -> origin/gh/dagitses/282/orig 2022-03-17T19:07:03.1318202Z * [new branch] gh/dagitses/283/base -> origin/gh/dagitses/283/base 2022-03-17T19:07:03.1319708Z * [new branch] gh/dagitses/283/orig -> origin/gh/dagitses/283/orig 2022-03-17T19:07:03.1323297Z * [new branch] gh/dagitses/284/base -> origin/gh/dagitses/284/base 2022-03-17T19:07:03.1325103Z * [new branch] gh/dagitses/284/orig -> origin/gh/dagitses/284/orig 2022-03-17T19:07:03.1328409Z * [new branch] gh/dagitses/285/base -> origin/gh/dagitses/285/base 2022-03-17T19:07:03.1329939Z * [new branch] gh/dagitses/285/orig -> origin/gh/dagitses/285/orig 2022-03-17T19:07:03.1333073Z * [new branch] gh/dagitses/286/base -> origin/gh/dagitses/286/base 2022-03-17T19:07:03.1335218Z * [new branch] gh/dagitses/286/orig -> origin/gh/dagitses/286/orig 2022-03-17T19:07:03.1338522Z * [new branch] gh/dagitses/287/base -> origin/gh/dagitses/287/base 2022-03-17T19:07:03.1341493Z * [new branch] gh/dagitses/287/head -> origin/gh/dagitses/287/head 2022-03-17T19:07:03.1344994Z * [new branch] gh/dagitses/287/orig -> origin/gh/dagitses/287/orig 2022-03-17T19:07:03.1347429Z * [new branch] gh/dagitses/288/base -> origin/gh/dagitses/288/base 2022-03-17T19:07:03.1349734Z * [new branch] gh/dagitses/288/head -> origin/gh/dagitses/288/head 2022-03-17T19:07:03.1351855Z * [new branch] gh/dagitses/288/orig -> origin/gh/dagitses/288/orig 2022-03-17T19:07:03.1355155Z * [new branch] gh/dagitses/289/base -> origin/gh/dagitses/289/base 2022-03-17T19:07:03.1356925Z * [new branch] gh/dagitses/289/head -> origin/gh/dagitses/289/head 2022-03-17T19:07:03.1358880Z * [new branch] gh/dagitses/289/orig -> origin/gh/dagitses/289/orig 2022-03-17T19:07:03.1362541Z * [new branch] gh/dagitses/29/base -> origin/gh/dagitses/29/base 2022-03-17T19:07:03.1364105Z * [new branch] gh/dagitses/29/head -> origin/gh/dagitses/29/head 2022-03-17T19:07:03.1366439Z * [new branch] gh/dagitses/29/orig -> origin/gh/dagitses/29/orig 2022-03-17T19:07:03.1369848Z * [new branch] gh/dagitses/290/base -> origin/gh/dagitses/290/base 2022-03-17T19:07:03.1371499Z * [new branch] gh/dagitses/290/head -> origin/gh/dagitses/290/head 2022-03-17T19:07:03.1373638Z * [new branch] gh/dagitses/290/orig -> origin/gh/dagitses/290/orig 2022-03-17T19:07:03.1377161Z * [new branch] gh/dagitses/3/base -> origin/gh/dagitses/3/base 2022-03-17T19:07:03.1378801Z * [new branch] gh/dagitses/3/orig -> origin/gh/dagitses/3/orig 2022-03-17T19:07:03.1382642Z * [new branch] gh/dagitses/30/base -> origin/gh/dagitses/30/base 2022-03-17T19:07:03.1384170Z * [new branch] gh/dagitses/30/head -> origin/gh/dagitses/30/head 2022-03-17T19:07:03.1386349Z * [new branch] gh/dagitses/30/orig -> origin/gh/dagitses/30/orig 2022-03-17T19:07:03.1389896Z * [new branch] gh/dagitses/31/base -> origin/gh/dagitses/31/base 2022-03-17T19:07:03.1391711Z * [new branch] gh/dagitses/31/head -> origin/gh/dagitses/31/head 2022-03-17T19:07:03.1393663Z * [new branch] gh/dagitses/31/orig -> origin/gh/dagitses/31/orig 2022-03-17T19:07:03.1397310Z * [new branch] gh/dagitses/32/base -> origin/gh/dagitses/32/base 2022-03-17T19:07:03.1399307Z * [new branch] gh/dagitses/32/head -> origin/gh/dagitses/32/head 2022-03-17T19:07:03.1401385Z * [new branch] gh/dagitses/32/orig -> origin/gh/dagitses/32/orig 2022-03-17T19:07:03.1404572Z * [new branch] gh/dagitses/33/base -> origin/gh/dagitses/33/base 2022-03-17T19:07:03.1406336Z * [new branch] gh/dagitses/33/head -> origin/gh/dagitses/33/head 2022-03-17T19:07:03.1409054Z * [new branch] gh/dagitses/33/orig -> origin/gh/dagitses/33/orig 2022-03-17T19:07:03.1412314Z * [new branch] gh/dagitses/34/base -> origin/gh/dagitses/34/base 2022-03-17T19:07:03.1414282Z * [new branch] gh/dagitses/34/head -> origin/gh/dagitses/34/head 2022-03-17T19:07:03.1416558Z * [new branch] gh/dagitses/34/orig -> origin/gh/dagitses/34/orig 2022-03-17T19:07:03.1419705Z * [new branch] gh/dagitses/35/base -> origin/gh/dagitses/35/base 2022-03-17T19:07:03.1422141Z * [new branch] gh/dagitses/35/head -> origin/gh/dagitses/35/head 2022-03-17T19:07:03.1425889Z * [new branch] gh/dagitses/35/orig -> origin/gh/dagitses/35/orig 2022-03-17T19:07:03.1428222Z * [new branch] gh/dagitses/36/base -> origin/gh/dagitses/36/base 2022-03-17T19:07:03.1430515Z * [new branch] gh/dagitses/36/head -> origin/gh/dagitses/36/head 2022-03-17T19:07:03.1432719Z * [new branch] gh/dagitses/36/orig -> origin/gh/dagitses/36/orig 2022-03-17T19:07:03.1435780Z * [new branch] gh/dagitses/37/base -> origin/gh/dagitses/37/base 2022-03-17T19:07:03.1437932Z * [new branch] gh/dagitses/37/head -> origin/gh/dagitses/37/head 2022-03-17T19:07:03.1440092Z * [new branch] gh/dagitses/37/orig -> origin/gh/dagitses/37/orig 2022-03-17T19:07:03.1442601Z * [new branch] gh/dagitses/38/base -> origin/gh/dagitses/38/base 2022-03-17T19:07:03.1444280Z * [new branch] gh/dagitses/38/head -> origin/gh/dagitses/38/head 2022-03-17T19:07:03.1446521Z * [new branch] gh/dagitses/38/orig -> origin/gh/dagitses/38/orig 2022-03-17T19:07:03.1449013Z * [new branch] gh/dagitses/39/base -> origin/gh/dagitses/39/base 2022-03-17T19:07:03.1451665Z * [new branch] gh/dagitses/39/head -> origin/gh/dagitses/39/head 2022-03-17T19:07:03.1453111Z * [new branch] gh/dagitses/39/orig -> origin/gh/dagitses/39/orig 2022-03-17T19:07:03.1456463Z * [new branch] gh/dagitses/4/base -> origin/gh/dagitses/4/base 2022-03-17T19:07:03.1458239Z * [new branch] gh/dagitses/4/orig -> origin/gh/dagitses/4/orig 2022-03-17T19:07:03.1461737Z * [new branch] gh/dagitses/40/base -> origin/gh/dagitses/40/base 2022-03-17T19:07:03.1464121Z * [new branch] gh/dagitses/40/head -> origin/gh/dagitses/40/head 2022-03-17T19:07:03.1466660Z * [new branch] gh/dagitses/40/orig -> origin/gh/dagitses/40/orig 2022-03-17T19:07:03.1469858Z * [new branch] gh/dagitses/41/base -> origin/gh/dagitses/41/base 2022-03-17T19:07:03.1471785Z * [new branch] gh/dagitses/41/head -> origin/gh/dagitses/41/head 2022-03-17T19:07:03.1473872Z * [new branch] gh/dagitses/41/orig -> origin/gh/dagitses/41/orig 2022-03-17T19:07:03.1477298Z * [new branch] gh/dagitses/42/base -> origin/gh/dagitses/42/base 2022-03-17T19:07:03.1478988Z * [new branch] gh/dagitses/42/head -> origin/gh/dagitses/42/head 2022-03-17T19:07:03.1482615Z * [new branch] gh/dagitses/42/orig -> origin/gh/dagitses/42/orig 2022-03-17T19:07:03.1484500Z * [new branch] gh/dagitses/43/base -> origin/gh/dagitses/43/base 2022-03-17T19:07:03.1486851Z * [new branch] gh/dagitses/43/head -> origin/gh/dagitses/43/head 2022-03-17T19:07:03.1489061Z * [new branch] gh/dagitses/43/orig -> origin/gh/dagitses/43/orig 2022-03-17T19:07:03.1492388Z * [new branch] gh/dagitses/44/base -> origin/gh/dagitses/44/base 2022-03-17T19:07:03.1494268Z * [new branch] gh/dagitses/44/head -> origin/gh/dagitses/44/head 2022-03-17T19:07:03.1496965Z * [new branch] gh/dagitses/44/orig -> origin/gh/dagitses/44/orig 2022-03-17T19:07:03.1500048Z * [new branch] gh/dagitses/45/base -> origin/gh/dagitses/45/base 2022-03-17T19:07:03.1502273Z * [new branch] gh/dagitses/45/head -> origin/gh/dagitses/45/head 2022-03-17T19:07:03.1504388Z * [new branch] gh/dagitses/45/orig -> origin/gh/dagitses/45/orig 2022-03-17T19:07:03.1507916Z * [new branch] gh/dagitses/46/base -> origin/gh/dagitses/46/base 2022-03-17T19:07:03.1509377Z * [new branch] gh/dagitses/46/orig -> origin/gh/dagitses/46/orig 2022-03-17T19:07:03.1513120Z * [new branch] gh/dagitses/47/base -> origin/gh/dagitses/47/base 2022-03-17T19:07:03.1514863Z * [new branch] gh/dagitses/47/head -> origin/gh/dagitses/47/head 2022-03-17T19:07:03.1517105Z * [new branch] gh/dagitses/47/orig -> origin/gh/dagitses/47/orig 2022-03-17T19:07:03.1520358Z * [new branch] gh/dagitses/5/base -> origin/gh/dagitses/5/base 2022-03-17T19:07:03.1522137Z * [new branch] gh/dagitses/5/head -> origin/gh/dagitses/5/head 2022-03-17T19:07:03.1524241Z * [new branch] gh/dagitses/5/orig -> origin/gh/dagitses/5/orig 2022-03-17T19:07:03.1527918Z * [new branch] gh/dagitses/51/base -> origin/gh/dagitses/51/base 2022-03-17T19:07:03.1529706Z * [new branch] gh/dagitses/51/orig -> origin/gh/dagitses/51/orig 2022-03-17T19:07:03.1532877Z * [new branch] gh/dagitses/52/base -> origin/gh/dagitses/52/base 2022-03-17T19:07:03.1534814Z * [new branch] gh/dagitses/52/head -> origin/gh/dagitses/52/head 2022-03-17T19:07:03.1537058Z * [new branch] gh/dagitses/52/orig -> origin/gh/dagitses/52/orig 2022-03-17T19:07:03.1540656Z * [new branch] gh/dagitses/53/base -> origin/gh/dagitses/53/base 2022-03-17T19:07:03.1542533Z * [new branch] gh/dagitses/53/head -> origin/gh/dagitses/53/head 2022-03-17T19:07:03.1545612Z * [new branch] gh/dagitses/54/base -> origin/gh/dagitses/54/base 2022-03-17T19:07:03.1548376Z * [new branch] gh/dagitses/55/base -> origin/gh/dagitses/55/base 2022-03-17T19:07:03.1551369Z * [new branch] gh/dagitses/56/base -> origin/gh/dagitses/56/base 2022-03-17T19:07:03.1554602Z * [new branch] gh/dagitses/57/base -> origin/gh/dagitses/57/base 2022-03-17T19:07:03.1556834Z * [new branch] gh/dagitses/58/base -> origin/gh/dagitses/58/base 2022-03-17T19:07:03.1558905Z * [new branch] gh/dagitses/58/head -> origin/gh/dagitses/58/head 2022-03-17T19:07:03.1562072Z * [new branch] gh/dagitses/59/base -> origin/gh/dagitses/59/base 2022-03-17T19:07:03.1564817Z * [new branch] gh/dagitses/60/base -> origin/gh/dagitses/60/base 2022-03-17T19:07:03.1567778Z * [new branch] gh/dagitses/61/base -> origin/gh/dagitses/61/base 2022-03-17T19:07:03.1570561Z * [new branch] gh/dagitses/62/base -> origin/gh/dagitses/62/base 2022-03-17T19:07:03.1573497Z * [new branch] gh/dagitses/63/base -> origin/gh/dagitses/63/base 2022-03-17T19:07:03.1575331Z * [new branch] gh/dagitses/63/head -> origin/gh/dagitses/63/head 2022-03-17T19:07:03.1578820Z * [new branch] gh/dagitses/64/base -> origin/gh/dagitses/64/base 2022-03-17T19:07:03.1580861Z * [new branch] gh/dagitses/64/head -> origin/gh/dagitses/64/head 2022-03-17T19:07:03.1583404Z * [new branch] gh/dagitses/64/orig -> origin/gh/dagitses/64/orig 2022-03-17T19:07:03.1585692Z * [new branch] gh/dagitses/65/base -> origin/gh/dagitses/65/base 2022-03-17T19:07:03.1587093Z * [new branch] gh/dagitses/65/head -> origin/gh/dagitses/65/head 2022-03-17T19:07:03.1589102Z * [new branch] gh/dagitses/65/orig -> origin/gh/dagitses/65/orig 2022-03-17T19:07:03.1591024Z * [new branch] gh/dagitses/66/base -> origin/gh/dagitses/66/base 2022-03-17T19:07:03.1592240Z * [new branch] gh/dagitses/66/head -> origin/gh/dagitses/66/head 2022-03-17T19:07:03.1593839Z * [new branch] gh/dagitses/66/orig -> origin/gh/dagitses/66/orig 2022-03-17T19:07:03.1595953Z * [new branch] gh/dagitses/67/base -> origin/gh/dagitses/67/base 2022-03-17T19:07:03.1597020Z * [new branch] gh/dagitses/67/head -> origin/gh/dagitses/67/head 2022-03-17T19:07:03.1598854Z * [new branch] gh/dagitses/67/orig -> origin/gh/dagitses/67/orig 2022-03-17T19:07:03.1600822Z * [new branch] gh/dagitses/68/base -> origin/gh/dagitses/68/base 2022-03-17T19:07:03.1601884Z * [new branch] gh/dagitses/68/head -> origin/gh/dagitses/68/head 2022-03-17T19:07:03.1603567Z * [new branch] gh/dagitses/68/orig -> origin/gh/dagitses/68/orig 2022-03-17T19:07:03.1605652Z * [new branch] gh/dagitses/69/base -> origin/gh/dagitses/69/base 2022-03-17T19:07:03.1606849Z * [new branch] gh/dagitses/69/head -> origin/gh/dagitses/69/head 2022-03-17T19:07:03.1608523Z * [new branch] gh/dagitses/69/orig -> origin/gh/dagitses/69/orig 2022-03-17T19:07:03.1610465Z * [new branch] gh/dagitses/7/base -> origin/gh/dagitses/7/base 2022-03-17T19:07:03.1611448Z * [new branch] gh/dagitses/7/orig -> origin/gh/dagitses/7/orig 2022-03-17T19:07:03.1613471Z * [new branch] gh/dagitses/70/base -> origin/gh/dagitses/70/base 2022-03-17T19:07:03.1614307Z * [new branch] gh/dagitses/70/head -> origin/gh/dagitses/70/head 2022-03-17T19:07:03.1615919Z * [new branch] gh/dagitses/70/orig -> origin/gh/dagitses/70/orig 2022-03-17T19:07:03.1617515Z * [new branch] gh/dagitses/71/base -> origin/gh/dagitses/71/base 2022-03-17T19:07:03.1618513Z * [new branch] gh/dagitses/71/head -> origin/gh/dagitses/71/head 2022-03-17T19:07:03.1620192Z * [new branch] gh/dagitses/71/orig -> origin/gh/dagitses/71/orig 2022-03-17T19:07:03.1622260Z * [new branch] gh/dagitses/72/base -> origin/gh/dagitses/72/base 2022-03-17T19:07:03.1623832Z * [new branch] gh/dagitses/72/head -> origin/gh/dagitses/72/head 2022-03-17T19:07:03.1624975Z * [new branch] gh/dagitses/72/orig -> origin/gh/dagitses/72/orig 2022-03-17T19:07:03.1627616Z * [new branch] gh/dagitses/73/base -> origin/gh/dagitses/73/base 2022-03-17T19:07:03.1628681Z * [new branch] gh/dagitses/73/head -> origin/gh/dagitses/73/head 2022-03-17T19:07:03.1630415Z * [new branch] gh/dagitses/73/orig -> origin/gh/dagitses/73/orig 2022-03-17T19:07:03.1632344Z * [new branch] gh/dagitses/74/base -> origin/gh/dagitses/74/base 2022-03-17T19:07:03.1633499Z * [new branch] gh/dagitses/74/head -> origin/gh/dagitses/74/head 2022-03-17T19:07:03.1635192Z * [new branch] gh/dagitses/74/orig -> origin/gh/dagitses/74/orig 2022-03-17T19:07:03.1636981Z * [new branch] gh/dagitses/75/base -> origin/gh/dagitses/75/base 2022-03-17T19:07:03.1637827Z * [new branch] gh/dagitses/75/head -> origin/gh/dagitses/75/head 2022-03-17T19:07:03.1639578Z * [new branch] gh/dagitses/75/orig -> origin/gh/dagitses/75/orig 2022-03-17T19:07:03.1641429Z * [new branch] gh/dagitses/76/base -> origin/gh/dagitses/76/base 2022-03-17T19:07:03.1642587Z * [new branch] gh/dagitses/76/head -> origin/gh/dagitses/76/head 2022-03-17T19:07:03.1644496Z * [new branch] gh/dagitses/76/orig -> origin/gh/dagitses/76/orig 2022-03-17T19:07:03.1659630Z * [new branch] gh/dagitses/77/base -> origin/gh/dagitses/77/base 2022-03-17T19:07:03.1660207Z * [new branch] gh/dagitses/77/head -> origin/gh/dagitses/77/head 2022-03-17T19:07:03.1661067Z * [new branch] gh/dagitses/77/orig -> origin/gh/dagitses/77/orig 2022-03-17T19:07:03.1661536Z * [new branch] gh/dagitses/78/base -> origin/gh/dagitses/78/base 2022-03-17T19:07:03.1662082Z * [new branch] gh/dagitses/78/orig -> origin/gh/dagitses/78/orig 2022-03-17T19:07:03.1662645Z * [new branch] gh/dagitses/79/base -> origin/gh/dagitses/79/base 2022-03-17T19:07:03.1663154Z * [new branch] gh/dagitses/79/head -> origin/gh/dagitses/79/head 2022-03-17T19:07:03.1663697Z * [new branch] gh/dagitses/79/orig -> origin/gh/dagitses/79/orig 2022-03-17T19:07:03.1664227Z * [new branch] gh/dagitses/8/base -> origin/gh/dagitses/8/base 2022-03-17T19:07:03.1664722Z * [new branch] gh/dagitses/8/orig -> origin/gh/dagitses/8/orig 2022-03-17T19:07:03.1665247Z * [new branch] gh/dagitses/80/base -> origin/gh/dagitses/80/base 2022-03-17T19:07:03.1665813Z * [new branch] gh/dagitses/80/head -> origin/gh/dagitses/80/head 2022-03-17T19:07:03.1666358Z * [new branch] gh/dagitses/80/orig -> origin/gh/dagitses/80/orig 2022-03-17T19:07:03.1667085Z * [new branch] gh/dagitses/81/base -> origin/gh/dagitses/81/base 2022-03-17T19:07:03.1667607Z * [new branch] gh/dagitses/81/head -> origin/gh/dagitses/81/head 2022-03-17T19:07:03.1668556Z * [new branch] gh/dagitses/82/base -> origin/gh/dagitses/82/base 2022-03-17T19:07:03.1669728Z * [new branch] gh/dagitses/82/head -> origin/gh/dagitses/82/head 2022-03-17T19:07:03.1672086Z * [new branch] gh/dagitses/83/base -> origin/gh/dagitses/83/base 2022-03-17T19:07:03.1673090Z * [new branch] gh/dagitses/83/head -> origin/gh/dagitses/83/head 2022-03-17T19:07:03.1674674Z * [new branch] gh/dagitses/84/base -> origin/gh/dagitses/84/base 2022-03-17T19:07:03.1676098Z * [new branch] gh/dagitses/84/head -> origin/gh/dagitses/84/head 2022-03-17T19:07:03.1677633Z * [new branch] gh/dagitses/85/base -> origin/gh/dagitses/85/base 2022-03-17T19:07:03.1679030Z * [new branch] gh/dagitses/85/head -> origin/gh/dagitses/85/head 2022-03-17T19:07:03.1680703Z * [new branch] gh/dagitses/86/base -> origin/gh/dagitses/86/base 2022-03-17T19:07:03.1682198Z * [new branch] gh/dagitses/86/head -> origin/gh/dagitses/86/head 2022-03-17T19:07:03.1683755Z * [new branch] gh/dagitses/87/base -> origin/gh/dagitses/87/base 2022-03-17T19:07:03.1685210Z * [new branch] gh/dagitses/87/head -> origin/gh/dagitses/87/head 2022-03-17T19:07:03.1686663Z * [new branch] gh/dagitses/88/base -> origin/gh/dagitses/88/base 2022-03-17T19:07:03.1687798Z * [new branch] gh/dagitses/88/head -> origin/gh/dagitses/88/head 2022-03-17T19:07:03.1689325Z * [new branch] gh/dagitses/89/base -> origin/gh/dagitses/89/base 2022-03-17T19:07:03.1690856Z * [new branch] gh/dagitses/89/head -> origin/gh/dagitses/89/head 2022-03-17T19:07:03.1693212Z * [new branch] gh/dagitses/9/base -> origin/gh/dagitses/9/base 2022-03-17T19:07:03.1694158Z * [new branch] gh/dagitses/9/orig -> origin/gh/dagitses/9/orig 2022-03-17T19:07:03.1696249Z * [new branch] gh/dagitses/90/base -> origin/gh/dagitses/90/base 2022-03-17T19:07:03.1697109Z * [new branch] gh/dagitses/90/head -> origin/gh/dagitses/90/head 2022-03-17T19:07:03.1699435Z * [new branch] gh/dagitses/91/base -> origin/gh/dagitses/91/base 2022-03-17T19:07:03.1699997Z * [new branch] gh/dagitses/91/head -> origin/gh/dagitses/91/head 2022-03-17T19:07:03.1702731Z * [new branch] gh/dagitses/92/base -> origin/gh/dagitses/92/base 2022-03-17T19:07:03.1703858Z * [new branch] gh/dagitses/92/head -> origin/gh/dagitses/92/head 2022-03-17T19:07:03.1705351Z * [new branch] gh/dagitses/93/base -> origin/gh/dagitses/93/base 2022-03-17T19:07:03.1706750Z * [new branch] gh/dagitses/93/head -> origin/gh/dagitses/93/head 2022-03-17T19:07:03.1708416Z * [new branch] gh/dagitses/94/base -> origin/gh/dagitses/94/base 2022-03-17T19:07:03.1709870Z * [new branch] gh/dagitses/94/head -> origin/gh/dagitses/94/head 2022-03-17T19:07:03.1711383Z * [new branch] gh/dagitses/95/base -> origin/gh/dagitses/95/base 2022-03-17T19:07:03.1712775Z * [new branch] gh/dagitses/95/head -> origin/gh/dagitses/95/head 2022-03-17T19:07:03.1714388Z * [new branch] gh/dagitses/96/base -> origin/gh/dagitses/96/base 2022-03-17T19:07:03.1715878Z * [new branch] gh/dagitses/96/head -> origin/gh/dagitses/96/head 2022-03-17T19:07:03.1717422Z * [new branch] gh/dagitses/97/base -> origin/gh/dagitses/97/base 2022-03-17T19:07:03.1718633Z * [new branch] gh/dagitses/97/head -> origin/gh/dagitses/97/head 2022-03-17T19:07:03.1720403Z * [new branch] gh/dagitses/98/base -> origin/gh/dagitses/98/base 2022-03-17T19:07:03.1721653Z * [new branch] gh/dagitses/98/head -> origin/gh/dagitses/98/head 2022-03-17T19:07:03.1723126Z * [new branch] gh/dagitses/99/base -> origin/gh/dagitses/99/base 2022-03-17T19:07:03.1724152Z * [new branch] gh/dagitses/99/head -> origin/gh/dagitses/99/head 2022-03-17T19:07:03.1726464Z * [new branch] gh/davidberard98/2/base -> origin/gh/davidberard98/2/base 2022-03-17T19:07:03.1727679Z * [new branch] gh/davidberard98/2/head -> origin/gh/davidberard98/2/head 2022-03-17T19:07:03.1729024Z * [new branch] gh/davidberard98/2/orig -> origin/gh/davidberard98/2/orig 2022-03-17T19:07:03.1730725Z * [new branch] gh/davidberard98/23/base -> origin/gh/davidberard98/23/base 2022-03-17T19:07:03.1732276Z * [new branch] gh/davidberard98/23/head -> origin/gh/davidberard98/23/head 2022-03-17T19:07:03.1733730Z * [new branch] gh/davidberard98/23/orig -> origin/gh/davidberard98/23/orig 2022-03-17T19:07:03.1735343Z * [new branch] gh/davidberard98/28/base -> origin/gh/davidberard98/28/base 2022-03-17T19:07:03.1741560Z * [new branch] gh/davidberard98/28/orig -> origin/gh/davidberard98/28/orig 2022-03-17T19:07:03.1742176Z * [new branch] gh/davidberard98/30/base -> origin/gh/davidberard98/30/base 2022-03-17T19:07:03.1742778Z * [new branch] gh/davidberard98/30/head -> origin/gh/davidberard98/30/head 2022-03-17T19:07:03.1743338Z * [new branch] gh/davidberard98/30/orig -> origin/gh/davidberard98/30/orig 2022-03-17T19:07:03.1743964Z * [new branch] gh/davidberard98/31/base -> origin/gh/davidberard98/31/base 2022-03-17T19:07:03.1744523Z * [new branch] gh/davidberard98/31/head -> origin/gh/davidberard98/31/head 2022-03-17T19:07:03.1745410Z * [new branch] gh/davidberard98/31/orig -> origin/gh/davidberard98/31/orig 2022-03-17T19:07:03.1747028Z * [new branch] gh/davidberard98/34/base -> origin/gh/davidberard98/34/base 2022-03-17T19:07:03.1748776Z * [new branch] gh/davidberard98/34/head -> origin/gh/davidberard98/34/head 2022-03-17T19:07:03.1763919Z * [new branch] gh/davidberard98/34/orig -> origin/gh/davidberard98/34/orig 2022-03-17T19:07:03.1764252Z * [new branch] gh/davidberard98/38/base -> origin/gh/davidberard98/38/base 2022-03-17T19:07:03.1764550Z * [new branch] gh/davidberard98/38/head -> origin/gh/davidberard98/38/head 2022-03-17T19:07:03.1764840Z * [new branch] gh/davidberard98/38/orig -> origin/gh/davidberard98/38/orig 2022-03-17T19:07:03.1765118Z * [new branch] gh/davidberard98/47/base -> origin/gh/davidberard98/47/base 2022-03-17T19:07:03.1765401Z * [new branch] gh/davidberard98/47/head -> origin/gh/davidberard98/47/head 2022-03-17T19:07:03.1765697Z * [new branch] gh/davidberard98/47/orig -> origin/gh/davidberard98/47/orig 2022-03-17T19:07:03.1765995Z * [new branch] gh/davidberard98/49/base -> origin/gh/davidberard98/49/base 2022-03-17T19:07:03.1766246Z * [new branch] gh/davidberard98/49/head -> origin/gh/davidberard98/49/head 2022-03-17T19:07:03.1766542Z * [new branch] gh/davidberard98/49/orig -> origin/gh/davidberard98/49/orig 2022-03-17T19:07:03.1766808Z * [new branch] gh/davidberard98/52/base -> origin/gh/davidberard98/52/base 2022-03-17T19:07:03.1767120Z * [new branch] gh/davidberard98/52/head -> origin/gh/davidberard98/52/head 2022-03-17T19:07:03.1768285Z * [new branch] gh/davidberard98/52/orig -> origin/gh/davidberard98/52/orig 2022-03-17T19:07:03.1770415Z * [new branch] gh/davidberard98/53/base -> origin/gh/davidberard98/53/base 2022-03-17T19:07:03.1771610Z * [new branch] gh/davidberard98/53/head -> origin/gh/davidberard98/53/head 2022-03-17T19:07:03.1773410Z * [new branch] gh/davidberard98/53/orig -> origin/gh/davidberard98/53/orig 2022-03-17T19:07:03.1775432Z * [new branch] gh/davidberard98/54/base -> origin/gh/davidberard98/54/base 2022-03-17T19:07:03.1776484Z * [new branch] gh/davidberard98/54/head -> origin/gh/davidberard98/54/head 2022-03-17T19:07:03.1778208Z * [new branch] gh/davidberard98/54/orig -> origin/gh/davidberard98/54/orig 2022-03-17T19:07:03.1779981Z * [new branch] gh/davidberard98/56/base -> origin/gh/davidberard98/56/base 2022-03-17T19:07:03.1781603Z * [new branch] gh/davidberard98/56/head -> origin/gh/davidberard98/56/head 2022-03-17T19:07:03.1783038Z * [new branch] gh/davidberard98/56/orig -> origin/gh/davidberard98/56/orig 2022-03-17T19:07:03.1784853Z * [new branch] gh/davidberard98/58/base -> origin/gh/davidberard98/58/base 2022-03-17T19:07:03.1786446Z * [new branch] gh/davidberard98/58/head -> origin/gh/davidberard98/58/head 2022-03-17T19:07:03.1788377Z * [new branch] gh/davidberard98/58/orig -> origin/gh/davidberard98/58/orig 2022-03-17T19:07:03.1790270Z * [new branch] gh/davidberard98/59/base -> origin/gh/davidberard98/59/base 2022-03-17T19:07:03.1791696Z * [new branch] gh/davidberard98/59/head -> origin/gh/davidberard98/59/head 2022-03-17T19:07:03.1793345Z * [new branch] gh/davidberard98/59/orig -> origin/gh/davidberard98/59/orig 2022-03-17T19:07:03.1794976Z * [new branch] gh/davidberard98/61/base -> origin/gh/davidberard98/61/base 2022-03-17T19:07:03.1796255Z * [new branch] gh/davidberard98/61/head -> origin/gh/davidberard98/61/head 2022-03-17T19:07:03.1797873Z * [new branch] gh/davidberard98/61/orig -> origin/gh/davidberard98/61/orig 2022-03-17T19:07:03.1800145Z * [new branch] gh/davidberard98/63/base -> origin/gh/davidberard98/63/base 2022-03-17T19:07:03.1800689Z * [new branch] gh/davidberard98/63/head -> origin/gh/davidberard98/63/head 2022-03-17T19:07:03.1802583Z * [new branch] gh/davidberard98/63/orig -> origin/gh/davidberard98/63/orig 2022-03-17T19:07:03.1803966Z * [new branch] gh/davidberard98/64/base -> origin/gh/davidberard98/64/base 2022-03-17T19:07:03.1806178Z * [new branch] gh/davidberard98/64/head -> origin/gh/davidberard98/64/head 2022-03-17T19:07:03.1809018Z * [new branch] gh/davidberard98/64/orig -> origin/gh/davidberard98/64/orig 2022-03-17T19:07:03.1809387Z * [new branch] gh/davidberard98/65/base -> origin/gh/davidberard98/65/base 2022-03-17T19:07:03.1811478Z * [new branch] gh/davidberard98/65/head -> origin/gh/davidberard98/65/head 2022-03-17T19:07:03.1812550Z * [new branch] gh/davidberard98/65/orig -> origin/gh/davidberard98/65/orig 2022-03-17T19:07:03.1814686Z * [new branch] gh/dbort/1/base -> origin/gh/dbort/1/base 2022-03-17T19:07:03.1816010Z * [new branch] gh/dbort/1/head -> origin/gh/dbort/1/head 2022-03-17T19:07:03.1817363Z * [new branch] gh/dbort/1/orig -> origin/gh/dbort/1/orig 2022-03-17T19:07:03.1819418Z * [new branch] gh/dbort/2/base -> origin/gh/dbort/2/base 2022-03-17T19:07:03.1820489Z * [new branch] gh/dbort/2/head -> origin/gh/dbort/2/head 2022-03-17T19:07:03.1822392Z * [new branch] gh/dbort/2/orig -> origin/gh/dbort/2/orig 2022-03-17T19:07:03.1824399Z * [new branch] gh/desertfire/18/base -> origin/gh/desertfire/18/base 2022-03-17T19:07:03.1825652Z * [new branch] gh/desertfire/18/head -> origin/gh/desertfire/18/head 2022-03-17T19:07:03.1827063Z * [new branch] gh/desertfire/18/orig -> origin/gh/desertfire/18/orig 2022-03-17T19:07:03.1829268Z * [new branch] gh/desertfire/20/base -> origin/gh/desertfire/20/base 2022-03-17T19:07:03.1830522Z * [new branch] gh/desertfire/20/head -> origin/gh/desertfire/20/head 2022-03-17T19:07:03.1831795Z * [new branch] gh/desertfire/20/orig -> origin/gh/desertfire/20/orig 2022-03-17T19:07:03.1833903Z * [new branch] gh/dhruvbird/20/base -> origin/gh/dhruvbird/20/base 2022-03-17T19:07:03.1835269Z * [new branch] gh/dhruvbird/20/head -> origin/gh/dhruvbird/20/head 2022-03-17T19:07:03.1836704Z * [new branch] gh/dhruvbird/20/orig -> origin/gh/dhruvbird/20/orig 2022-03-17T19:07:03.1838930Z * [new branch] gh/dhruvbird/23/base -> origin/gh/dhruvbird/23/base 2022-03-17T19:07:03.1840325Z * [new branch] gh/dhruvbird/23/head -> origin/gh/dhruvbird/23/head 2022-03-17T19:07:03.1841728Z * [new branch] gh/dhruvbird/23/orig -> origin/gh/dhruvbird/23/orig 2022-03-17T19:07:03.1843836Z * [new branch] gh/dhruvbird/31/base -> origin/gh/dhruvbird/31/base 2022-03-17T19:07:03.1844728Z * [new branch] gh/dhruvbird/31/head -> origin/gh/dhruvbird/31/head 2022-03-17T19:07:03.1846170Z * [new branch] gh/dhruvbird/31/orig -> origin/gh/dhruvbird/31/orig 2022-03-17T19:07:03.1848313Z * [new branch] gh/dhruvbird/52/base -> origin/gh/dhruvbird/52/base 2022-03-17T19:07:03.1849557Z * [new branch] gh/dhruvbird/52/head -> origin/gh/dhruvbird/52/head 2022-03-17T19:07:03.1850997Z * [new branch] gh/dhruvbird/52/orig -> origin/gh/dhruvbird/52/orig 2022-03-17T19:07:03.1853123Z * [new branch] gh/dhruvbird/55/base -> origin/gh/dhruvbird/55/base 2022-03-17T19:07:03.1854465Z * [new branch] gh/dhruvbird/55/head -> origin/gh/dhruvbird/55/head 2022-03-17T19:07:03.1856399Z * [new branch] gh/dhruvbird/55/orig -> origin/gh/dhruvbird/55/orig 2022-03-17T19:07:03.1857860Z * [new branch] gh/dhruvbird/56/base -> origin/gh/dhruvbird/56/base 2022-03-17T19:07:03.1859316Z * [new branch] gh/dhruvbird/56/head -> origin/gh/dhruvbird/56/head 2022-03-17T19:07:03.1861095Z * [new branch] gh/dhruvbird/57/base -> origin/gh/dhruvbird/57/base 2022-03-17T19:07:03.1862302Z * [new branch] gh/dhruvbird/57/head -> origin/gh/dhruvbird/57/head 2022-03-17T19:07:03.1863706Z * [new branch] gh/dhruvbird/58/base -> origin/gh/dhruvbird/58/base 2022-03-17T19:07:03.1864975Z * [new branch] gh/dhruvbird/58/head -> origin/gh/dhruvbird/58/head 2022-03-17T19:07:03.1867119Z * [new branch] gh/dhruvbird/59/base -> origin/gh/dhruvbird/59/base 2022-03-17T19:07:03.1867743Z * [new branch] gh/dhruvbird/59/head -> origin/gh/dhruvbird/59/head 2022-03-17T19:07:03.1869647Z * [new branch] gh/dhruvbird/60/base -> origin/gh/dhruvbird/60/base 2022-03-17T19:07:03.1870534Z * [new branch] gh/dhruvbird/60/head -> origin/gh/dhruvbird/60/head 2022-03-17T19:07:03.1872426Z * [new branch] gh/dhruvbird/61/base -> origin/gh/dhruvbird/61/base 2022-03-17T19:07:03.1873484Z * [new branch] gh/dhruvbird/61/head -> origin/gh/dhruvbird/61/head 2022-03-17T19:07:03.1875684Z * [new branch] gh/dhruvbird/66/base -> origin/gh/dhruvbird/66/base 2022-03-17T19:07:03.1876695Z * [new branch] gh/dhruvbird/66/head -> origin/gh/dhruvbird/66/head 2022-03-17T19:07:03.1878167Z * [new branch] gh/dhruvbird/66/orig -> origin/gh/dhruvbird/66/orig 2022-03-17T19:07:03.1880089Z * [new branch] gh/dhruvbird/75/base -> origin/gh/dhruvbird/75/base 2022-03-17T19:07:03.1881377Z * [new branch] gh/dhruvbird/75/head -> origin/gh/dhruvbird/75/head 2022-03-17T19:07:03.1883285Z * [new branch] gh/dhruvbird/75/orig -> origin/gh/dhruvbird/75/orig 2022-03-17T19:07:03.1885009Z * [new branch] gh/dhruvbird/87/base -> origin/gh/dhruvbird/87/base 2022-03-17T19:07:03.1886286Z * [new branch] gh/dhruvbird/87/head -> origin/gh/dhruvbird/87/head 2022-03-17T19:07:03.1887985Z * [new branch] gh/dhruvbird/88/base -> origin/gh/dhruvbird/88/base 2022-03-17T19:07:03.1889014Z * [new branch] gh/dhruvbird/88/head -> origin/gh/dhruvbird/88/head 2022-03-17T19:07:03.1891249Z * [new branch] gh/dhruvbird/92/base -> origin/gh/dhruvbird/92/base 2022-03-17T19:07:03.1892532Z * [new branch] gh/dhruvbird/92/head -> origin/gh/dhruvbird/92/head 2022-03-17T19:07:03.1894086Z * [new branch] gh/dhruvbird/92/orig -> origin/gh/dhruvbird/92/orig 2022-03-17T19:07:03.1895846Z * [new branch] gh/dhruvbird/93/base -> origin/gh/dhruvbird/93/base 2022-03-17T19:07:03.1897083Z * [new branch] gh/dhruvbird/93/head -> origin/gh/dhruvbird/93/head 2022-03-17T19:07:03.1898402Z * [new branch] gh/dhruvbird/93/orig -> origin/gh/dhruvbird/93/orig 2022-03-17T19:07:03.1900110Z * [new branch] gh/dhruvbird/94/base -> origin/gh/dhruvbird/94/base 2022-03-17T19:07:03.1901195Z * [new branch] gh/dhruvbird/94/head -> origin/gh/dhruvbird/94/head 2022-03-17T19:07:03.1902987Z * [new branch] gh/dhruvbird/94/orig -> origin/gh/dhruvbird/94/orig 2022-03-17T19:07:03.1905114Z * [new branch] gh/dhruvbird/95/base -> origin/gh/dhruvbird/95/base 2022-03-17T19:07:03.1906266Z * [new branch] gh/dhruvbird/95/head -> origin/gh/dhruvbird/95/head 2022-03-17T19:07:03.1907806Z * [new branch] gh/dhruvbird/95/orig -> origin/gh/dhruvbird/95/orig 2022-03-17T19:07:03.1909888Z * [new branch] gh/dhruvbird/96/base -> origin/gh/dhruvbird/96/base 2022-03-17T19:07:03.1910763Z * [new branch] gh/dhruvbird/96/head -> origin/gh/dhruvbird/96/head 2022-03-17T19:07:03.1912923Z * [new branch] gh/dhruvbird/96/orig -> origin/gh/dhruvbird/96/orig 2022-03-17T19:07:03.1915052Z * [new branch] gh/dhruvbird/97/base -> origin/gh/dhruvbird/97/base 2022-03-17T19:07:03.1915835Z * [new branch] gh/dhruvbird/97/head -> origin/gh/dhruvbird/97/head 2022-03-17T19:07:03.1917755Z * [new branch] gh/dhruvbird/97/orig -> origin/gh/dhruvbird/97/orig 2022-03-17T19:07:03.1919671Z * [new branch] gh/dracifer/1/base -> origin/gh/dracifer/1/base 2022-03-17T19:07:03.1920593Z * [new branch] gh/dracifer/1/head -> origin/gh/dracifer/1/head 2022-03-17T19:07:03.1922521Z * [new branch] gh/dracifer/1/orig -> origin/gh/dracifer/1/orig 2022-03-17T19:07:03.1923800Z * [new branch] gh/dracifer/2/base -> origin/gh/dracifer/2/base 2022-03-17T19:07:03.1925422Z * [new branch] gh/dracifer/2/orig -> origin/gh/dracifer/2/orig 2022-03-17T19:07:03.1927422Z * [new branch] gh/drdarshan/1/base -> origin/gh/drdarshan/1/base 2022-03-17T19:07:03.1928692Z * [new branch] gh/drdarshan/1/head -> origin/gh/drdarshan/1/head 2022-03-17T19:07:03.1930066Z * [new branch] gh/drdarshan/1/orig -> origin/gh/drdarshan/1/orig 2022-03-17T19:07:03.1932003Z * [new branch] gh/drdarshan/2/base -> origin/gh/drdarshan/2/base 2022-03-17T19:07:03.1933047Z * [new branch] gh/drdarshan/2/orig -> origin/gh/drdarshan/2/orig 2022-03-17T19:07:03.1934895Z * [new branch] gh/drdarshan/3/base -> origin/gh/drdarshan/3/base 2022-03-17T19:07:03.1936259Z * [new branch] gh/drdarshan/3/orig -> origin/gh/drdarshan/3/orig 2022-03-17T19:07:03.1938239Z * [new branch] gh/dreiss/1/base -> origin/gh/dreiss/1/base 2022-03-17T19:07:03.1940196Z * [new branch] gh/dreiss/1/head -> origin/gh/dreiss/1/head 2022-03-17T19:07:03.1941197Z * [new branch] gh/dreiss/1/orig -> origin/gh/dreiss/1/orig 2022-03-17T19:07:03.1943549Z * [new branch] gh/dreiss/10/base -> origin/gh/dreiss/10/base 2022-03-17T19:07:03.1944619Z * [new branch] gh/dreiss/10/head -> origin/gh/dreiss/10/head 2022-03-17T19:07:03.1946082Z * [new branch] gh/dreiss/10/orig -> origin/gh/dreiss/10/orig 2022-03-17T19:07:03.1948000Z * [new branch] gh/dreiss/105/base -> origin/gh/dreiss/105/base 2022-03-17T19:07:03.1949100Z * [new branch] gh/dreiss/105/head -> origin/gh/dreiss/105/head 2022-03-17T19:07:03.1950976Z * [new branch] gh/dreiss/105/orig -> origin/gh/dreiss/105/orig 2022-03-17T19:07:03.1953021Z * [new branch] gh/dreiss/106/base -> origin/gh/dreiss/106/base 2022-03-17T19:07:03.1954680Z * [new branch] gh/dreiss/106/head -> origin/gh/dreiss/106/head 2022-03-17T19:07:03.1955794Z * [new branch] gh/dreiss/106/orig -> origin/gh/dreiss/106/orig 2022-03-17T19:07:03.1958075Z * [new branch] gh/dreiss/109/base -> origin/gh/dreiss/109/base 2022-03-17T19:07:03.1959262Z * [new branch] gh/dreiss/109/head -> origin/gh/dreiss/109/head 2022-03-17T19:07:03.1960890Z * [new branch] gh/dreiss/109/orig -> origin/gh/dreiss/109/orig 2022-03-17T19:07:03.1963240Z * [new branch] gh/dreiss/110/base -> origin/gh/dreiss/110/base 2022-03-17T19:07:03.1964481Z * [new branch] gh/dreiss/110/head -> origin/gh/dreiss/110/head 2022-03-17T19:07:03.1965935Z * [new branch] gh/dreiss/110/orig -> origin/gh/dreiss/110/orig 2022-03-17T19:07:03.1967297Z * [new branch] gh/dreiss/2/base -> origin/gh/dreiss/2/base 2022-03-17T19:07:03.1968965Z * [new branch] gh/dreiss/2/head -> origin/gh/dreiss/2/head 2022-03-17T19:07:03.1970065Z * [new branch] gh/dreiss/2/orig -> origin/gh/dreiss/2/orig 2022-03-17T19:07:03.1972306Z * [new branch] gh/dreiss/25/base -> origin/gh/dreiss/25/base 2022-03-17T19:07:03.1973198Z * [new branch] gh/dreiss/25/head -> origin/gh/dreiss/25/head 2022-03-17T19:07:03.1974862Z * [new branch] gh/dreiss/25/orig -> origin/gh/dreiss/25/orig 2022-03-17T19:07:03.1976775Z * [new branch] gh/dreiss/28/base -> origin/gh/dreiss/28/base 2022-03-17T19:07:03.1978002Z * [new branch] gh/dreiss/28/head -> origin/gh/dreiss/28/head 2022-03-17T19:07:03.1979874Z * [new branch] gh/dreiss/28/orig -> origin/gh/dreiss/28/orig 2022-03-17T19:07:03.1982642Z * [new branch] gh/dreiss/33/base -> origin/gh/dreiss/33/base 2022-03-17T19:07:03.1984051Z * [new branch] gh/dreiss/33/head -> origin/gh/dreiss/33/head 2022-03-17T19:07:03.1985491Z * [new branch] gh/dreiss/33/orig -> origin/gh/dreiss/33/orig 2022-03-17T19:07:03.1987386Z * [new branch] gh/dreiss/36/base -> origin/gh/dreiss/36/base 2022-03-17T19:07:03.1988119Z * [new branch] gh/dreiss/36/head -> origin/gh/dreiss/36/head 2022-03-17T19:07:03.1990011Z * [new branch] gh/dreiss/36/orig -> origin/gh/dreiss/36/orig 2022-03-17T19:07:03.1991322Z * [new branch] gh/dreiss/39/base -> origin/gh/dreiss/39/base 2022-03-17T19:07:03.1992997Z * [new branch] gh/dreiss/39/head -> origin/gh/dreiss/39/head 2022-03-17T19:07:03.1994158Z * [new branch] gh/dreiss/39/orig -> origin/gh/dreiss/39/orig 2022-03-17T19:07:03.1996108Z * [new branch] gh/dreiss/43/base -> origin/gh/dreiss/43/base 2022-03-17T19:07:03.1997414Z * [new branch] gh/dreiss/43/head -> origin/gh/dreiss/43/head 2022-03-17T19:07:03.1998850Z * [new branch] gh/dreiss/43/orig -> origin/gh/dreiss/43/orig 2022-03-17T19:07:03.2001023Z * [new branch] gh/dreiss/63/base -> origin/gh/dreiss/63/base 2022-03-17T19:07:03.2002321Z * [new branch] gh/dreiss/63/head -> origin/gh/dreiss/63/head 2022-03-17T19:07:03.2003777Z * [new branch] gh/dreiss/63/orig -> origin/gh/dreiss/63/orig 2022-03-17T19:07:03.2005744Z * [new branch] gh/dreiss/67/base -> origin/gh/dreiss/67/base 2022-03-17T19:07:03.2006781Z * [new branch] gh/dreiss/67/head -> origin/gh/dreiss/67/head 2022-03-17T19:07:03.2008266Z * [new branch] gh/dreiss/67/orig -> origin/gh/dreiss/67/orig 2022-03-17T19:07:03.2009667Z * [new branch] gh/dreiss/68/base -> origin/gh/dreiss/68/base 2022-03-17T19:07:03.2011075Z * [new branch] gh/dreiss/68/head -> origin/gh/dreiss/68/head 2022-03-17T19:07:03.2012457Z * [new branch] gh/dreiss/68/orig -> origin/gh/dreiss/68/orig 2022-03-17T19:07:03.2014613Z * [new branch] gh/dreiss/71/base -> origin/gh/dreiss/71/base 2022-03-17T19:07:03.2015577Z * [new branch] gh/dreiss/71/head -> origin/gh/dreiss/71/head 2022-03-17T19:07:03.2017014Z * [new branch] gh/dreiss/71/orig -> origin/gh/dreiss/71/orig 2022-03-17T19:07:03.2018478Z * [new branch] gh/dreiss/72/base -> origin/gh/dreiss/72/base 2022-03-17T19:07:03.2020222Z * [new branch] gh/dreiss/72/head -> origin/gh/dreiss/72/head 2022-03-17T19:07:03.2021525Z * [new branch] gh/dreiss/72/orig -> origin/gh/dreiss/72/orig 2022-03-17T19:07:03.2024781Z * [new branch] gh/dreiss/73/base -> origin/gh/dreiss/73/base 2022-03-17T19:07:03.2026032Z * [new branch] gh/dreiss/73/head -> origin/gh/dreiss/73/head 2022-03-17T19:07:03.2027494Z * [new branch] gh/dreiss/73/orig -> origin/gh/dreiss/73/orig 2022-03-17T19:07:03.2029655Z * [new branch] gh/dreiss/75/base -> origin/gh/dreiss/75/base 2022-03-17T19:07:03.2030961Z * [new branch] gh/dreiss/75/head -> origin/gh/dreiss/75/head 2022-03-17T19:07:03.2033014Z * [new branch] gh/dreiss/80/base -> origin/gh/dreiss/80/base 2022-03-17T19:07:03.2034464Z * [new branch] gh/dreiss/80/head -> origin/gh/dreiss/80/head 2022-03-17T19:07:03.2035817Z * [new branch] gh/dreiss/80/orig -> origin/gh/dreiss/80/orig 2022-03-17T19:07:03.2038010Z * [new branch] gh/dskhudia/20/base -> origin/gh/dskhudia/20/base 2022-03-17T19:07:03.2039389Z * [new branch] gh/dskhudia/20/head -> origin/gh/dskhudia/20/head 2022-03-17T19:07:03.2041740Z * [new branch] gh/dskhudia/21/base -> origin/gh/dskhudia/21/base 2022-03-17T19:07:03.2043058Z * [new branch] gh/dskhudia/21/head -> origin/gh/dskhudia/21/head 2022-03-17T19:07:03.2044547Z * [new branch] gh/dskhudia/22/base -> origin/gh/dskhudia/22/base 2022-03-17T19:07:03.2045882Z * [new branch] gh/dskhudia/22/head -> origin/gh/dskhudia/22/head 2022-03-17T19:07:03.2047392Z * [new branch] gh/dskhudia/23/base -> origin/gh/dskhudia/23/base 2022-03-17T19:07:03.2048739Z * [new branch] gh/dskhudia/23/head -> origin/gh/dskhudia/23/head 2022-03-17T19:07:03.2050201Z * [new branch] gh/dskhudia/24/base -> origin/gh/dskhudia/24/base 2022-03-17T19:07:03.2051546Z * [new branch] gh/dskhudia/24/head -> origin/gh/dskhudia/24/head 2022-03-17T19:07:03.2053570Z * [new branch] gh/dskhudia/27/base -> origin/gh/dskhudia/27/base 2022-03-17T19:07:03.2054770Z * [new branch] gh/dskhudia/27/head -> origin/gh/dskhudia/27/head 2022-03-17T19:07:03.2056132Z * [new branch] gh/dskhudia/27/orig -> origin/gh/dskhudia/27/orig 2022-03-17T19:07:03.2059179Z * [new branch] gh/dzdang/1/base -> origin/gh/dzdang/1/base 2022-03-17T19:07:03.2060657Z * [new branch] gh/dzdang/1/orig -> origin/gh/dzdang/1/orig 2022-03-17T19:07:03.2062896Z * [new branch] gh/dzdang/10/base -> origin/gh/dzdang/10/base 2022-03-17T19:07:03.2066903Z * [new branch] gh/dzdang/10/orig -> origin/gh/dzdang/10/orig 2022-03-17T19:07:03.2067195Z * [new branch] gh/dzdang/11/base -> origin/gh/dzdang/11/base 2022-03-17T19:07:03.2067518Z * [new branch] gh/dzdang/11/orig -> origin/gh/dzdang/11/orig 2022-03-17T19:07:03.2068713Z * [new branch] gh/dzdang/13/base -> origin/gh/dzdang/13/base 2022-03-17T19:07:03.2070213Z * [new branch] gh/dzdang/13/head -> origin/gh/dzdang/13/head 2022-03-17T19:07:03.2071715Z * [new branch] gh/dzdang/13/orig -> origin/gh/dzdang/13/orig 2022-03-17T19:07:03.2073178Z * [new branch] gh/dzdang/14/base -> origin/gh/dzdang/14/base 2022-03-17T19:07:03.2074682Z * [new branch] gh/dzdang/14/head -> origin/gh/dzdang/14/head 2022-03-17T19:07:03.2075858Z * [new branch] gh/dzdang/14/orig -> origin/gh/dzdang/14/orig 2022-03-17T19:07:03.2078063Z * [new branch] gh/dzdang/15/base -> origin/gh/dzdang/15/base 2022-03-17T19:07:03.2079248Z * [new branch] gh/dzdang/15/head -> origin/gh/dzdang/15/head 2022-03-17T19:07:03.2080720Z * [new branch] gh/dzdang/15/orig -> origin/gh/dzdang/15/orig 2022-03-17T19:07:03.2082762Z * [new branch] gh/dzdang/2/base -> origin/gh/dzdang/2/base 2022-03-17T19:07:03.2083963Z * [new branch] gh/dzdang/2/orig -> origin/gh/dzdang/2/orig 2022-03-17T19:07:03.2086044Z * [new branch] gh/dzdang/29/base -> origin/gh/dzdang/29/base 2022-03-17T19:07:03.2087319Z * [new branch] gh/dzdang/29/head -> origin/gh/dzdang/29/head 2022-03-17T19:07:03.2088776Z * [new branch] gh/dzdang/29/orig -> origin/gh/dzdang/29/orig 2022-03-17T19:07:03.2090972Z * [new branch] gh/dzdang/30/base -> origin/gh/dzdang/30/base 2022-03-17T19:07:03.2092179Z * [new branch] gh/dzdang/30/head -> origin/gh/dzdang/30/head 2022-03-17T19:07:03.2093554Z * [new branch] gh/dzdang/30/orig -> origin/gh/dzdang/30/orig 2022-03-17T19:07:03.2095536Z * [new branch] gh/dzdang/36/orig -> origin/gh/dzdang/36/orig 2022-03-17T19:07:03.2097863Z * [new branch] gh/dzdang/37/base -> origin/gh/dzdang/37/base 2022-03-17T19:07:03.2098695Z * [new branch] gh/dzdang/37/head -> origin/gh/dzdang/37/head 2022-03-17T19:07:03.2100494Z * [new branch] gh/dzdang/37/orig -> origin/gh/dzdang/37/orig 2022-03-17T19:07:03.2102065Z * [new branch] gh/dzdang/38/base -> origin/gh/dzdang/38/base 2022-03-17T19:07:03.2103503Z * [new branch] gh/dzdang/38/head -> origin/gh/dzdang/38/head 2022-03-17T19:07:03.2104881Z * [new branch] gh/dzdang/38/orig -> origin/gh/dzdang/38/orig 2022-03-17T19:07:03.2107045Z * [new branch] gh/dzdang/4/base -> origin/gh/dzdang/4/base 2022-03-17T19:07:03.2107928Z * [new branch] gh/dzdang/4/orig -> origin/gh/dzdang/4/orig 2022-03-17T19:07:03.2110164Z * [new branch] gh/dzdang/42/base -> origin/gh/dzdang/42/base 2022-03-17T19:07:03.2111761Z * [new branch] gh/dzdang/42/head -> origin/gh/dzdang/42/head 2022-03-17T19:07:03.2113543Z * [new branch] gh/dzdang/42/orig -> origin/gh/dzdang/42/orig 2022-03-17T19:07:03.2115643Z * [new branch] gh/dzdang/47/base -> origin/gh/dzdang/47/base 2022-03-17T19:07:03.2117647Z * [new branch] gh/dzdang/47/head -> origin/gh/dzdang/47/head 2022-03-17T19:07:03.2119730Z * [new branch] gh/dzdang/47/orig -> origin/gh/dzdang/47/orig 2022-03-17T19:07:03.2122076Z * [new branch] gh/dzdang/48/base -> origin/gh/dzdang/48/base 2022-03-17T19:07:03.2123686Z * [new branch] gh/dzdang/48/head -> origin/gh/dzdang/48/head 2022-03-17T19:07:03.2125005Z * [new branch] gh/dzdang/48/orig -> origin/gh/dzdang/48/orig 2022-03-17T19:07:03.2126971Z * [new branch] gh/dzdang/49/base -> origin/gh/dzdang/49/base 2022-03-17T19:07:03.2128269Z * [new branch] gh/dzdang/49/head -> origin/gh/dzdang/49/head 2022-03-17T19:07:03.2129884Z * [new branch] gh/dzdang/49/orig -> origin/gh/dzdang/49/orig 2022-03-17T19:07:03.2131785Z * [new branch] gh/dzdang/5/base -> origin/gh/dzdang/5/base 2022-03-17T19:07:03.2132854Z * [new branch] gh/dzdang/5/orig -> origin/gh/dzdang/5/orig 2022-03-17T19:07:03.2134829Z * [new branch] gh/dzdang/50/base -> origin/gh/dzdang/50/base 2022-03-17T19:07:03.2136188Z * [new branch] gh/dzdang/50/head -> origin/gh/dzdang/50/head 2022-03-17T19:07:03.2138105Z * [new branch] gh/dzdang/50/orig -> origin/gh/dzdang/50/orig 2022-03-17T19:07:03.2141523Z * [new branch] gh/dzdang/51/base -> origin/gh/dzdang/51/base 2022-03-17T19:07:03.2142065Z * [new branch] gh/dzdang/51/head -> origin/gh/dzdang/51/head 2022-03-17T19:07:03.2142522Z * [new branch] gh/dzdang/51/orig -> origin/gh/dzdang/51/orig 2022-03-17T19:07:03.2144466Z * [new branch] gh/dzdang/52/base -> origin/gh/dzdang/52/base 2022-03-17T19:07:03.2146586Z * [new branch] gh/dzdang/52/head -> origin/gh/dzdang/52/head 2022-03-17T19:07:03.2146874Z * [new branch] gh/dzdang/52/orig -> origin/gh/dzdang/52/orig 2022-03-17T19:07:03.2148870Z * [new branch] gh/dzdang/53/base -> origin/gh/dzdang/53/base 2022-03-17T19:07:03.2150216Z * [new branch] gh/dzdang/53/head -> origin/gh/dzdang/53/head 2022-03-17T19:07:03.2151512Z * [new branch] gh/dzdang/53/orig -> origin/gh/dzdang/53/orig 2022-03-17T19:07:03.2153478Z * [new branch] gh/dzdang/54/base -> origin/gh/dzdang/54/base 2022-03-17T19:07:03.2155218Z * [new branch] gh/dzdang/54/head -> origin/gh/dzdang/54/head 2022-03-17T19:07:03.2156166Z * [new branch] gh/dzdang/54/orig -> origin/gh/dzdang/54/orig 2022-03-17T19:07:03.2158090Z * [new branch] gh/dzdang/55/base -> origin/gh/dzdang/55/base 2022-03-17T19:07:03.2159694Z * [new branch] gh/dzdang/55/head -> origin/gh/dzdang/55/head 2022-03-17T19:07:03.2161453Z * [new branch] gh/dzdang/55/orig -> origin/gh/dzdang/55/orig 2022-03-17T19:07:03.2163292Z * [new branch] gh/dzdang/56/base -> origin/gh/dzdang/56/base 2022-03-17T19:07:03.2164477Z * [new branch] gh/dzdang/56/head -> origin/gh/dzdang/56/head 2022-03-17T19:07:03.2165949Z * [new branch] gh/dzdang/56/orig -> origin/gh/dzdang/56/orig 2022-03-17T19:07:03.2167906Z * [new branch] gh/dzdang/57/base -> origin/gh/dzdang/57/base 2022-03-17T19:07:03.2169142Z * [new branch] gh/dzdang/57/head -> origin/gh/dzdang/57/head 2022-03-17T19:07:03.2170814Z * [new branch] gh/dzdang/57/orig -> origin/gh/dzdang/57/orig 2022-03-17T19:07:03.2172708Z * [new branch] gh/dzdang/58/base -> origin/gh/dzdang/58/base 2022-03-17T19:07:03.2174489Z * [new branch] gh/dzdang/58/head -> origin/gh/dzdang/58/head 2022-03-17T19:07:03.2176371Z * [new branch] gh/dzdang/58/orig -> origin/gh/dzdang/58/orig 2022-03-17T19:07:03.2178391Z * [new branch] gh/dzdang/59/base -> origin/gh/dzdang/59/base 2022-03-17T19:07:03.2179612Z * [new branch] gh/dzdang/59/head -> origin/gh/dzdang/59/head 2022-03-17T19:07:03.2181163Z * [new branch] gh/dzdang/59/orig -> origin/gh/dzdang/59/orig 2022-03-17T19:07:03.2183543Z * [new branch] gh/dzdang/6/base -> origin/gh/dzdang/6/base 2022-03-17T19:07:03.2184864Z * [new branch] gh/dzdang/6/orig -> origin/gh/dzdang/6/orig 2022-03-17T19:07:03.2186693Z * [new branch] gh/dzdang/7/base -> origin/gh/dzdang/7/base 2022-03-17T19:07:03.2187676Z * [new branch] gh/dzdang/7/orig -> origin/gh/dzdang/7/orig 2022-03-17T19:07:03.2189695Z * [new branch] gh/dzdang/8/base -> origin/gh/dzdang/8/base 2022-03-17T19:07:03.2191381Z * [new branch] gh/dzdang/8/orig -> origin/gh/dzdang/8/orig 2022-03-17T19:07:03.2193477Z * [new branch] gh/dzdang/9/base -> origin/gh/dzdang/9/base 2022-03-17T19:07:03.2194318Z * [new branch] gh/dzdang/9/orig -> origin/gh/dzdang/9/orig 2022-03-17T19:07:03.2196615Z * [new branch] gh/edmundw314/15/base -> origin/gh/edmundw314/15/base 2022-03-17T19:07:03.2198011Z * [new branch] gh/edmundw314/15/head -> origin/gh/edmundw314/15/head 2022-03-17T19:07:03.2199918Z * [new branch] gh/edmundw314/15/orig -> origin/gh/edmundw314/15/orig 2022-03-17T19:07:03.2201613Z * [new branch] gh/edmundw314/17/base -> origin/gh/edmundw314/17/base 2022-03-17T19:07:03.2202898Z * [new branch] gh/edmundw314/17/head -> origin/gh/edmundw314/17/head 2022-03-17T19:07:03.2204352Z * [new branch] gh/edmundw314/17/orig -> origin/gh/edmundw314/17/orig 2022-03-17T19:07:03.2206415Z * [new branch] gh/edmundw314/4/base -> origin/gh/edmundw314/4/base 2022-03-17T19:07:03.2207649Z * [new branch] gh/edmundw314/4/head -> origin/gh/edmundw314/4/head 2022-03-17T19:07:03.2209863Z * [new branch] gh/edmundw314/4/orig -> origin/gh/edmundw314/4/orig 2022-03-17T19:07:03.2211798Z * [new branch] gh/edmundw314/7/base -> origin/gh/edmundw314/7/base 2022-03-17T19:07:03.2212814Z * [new branch] gh/edmundw314/7/head -> origin/gh/edmundw314/7/head 2022-03-17T19:07:03.2214272Z * [new branch] gh/edmundw314/7/orig -> origin/gh/edmundw314/7/orig 2022-03-17T19:07:03.2216592Z * [new branch] gh/eellison/110/base -> origin/gh/eellison/110/base 2022-03-17T19:07:03.2218036Z * [new branch] gh/eellison/110/head -> origin/gh/eellison/110/head 2022-03-17T19:07:03.2220009Z * [new branch] gh/eellison/110/orig -> origin/gh/eellison/110/orig 2022-03-17T19:07:03.2221428Z * [new branch] gh/eellison/111/base -> origin/gh/eellison/111/base 2022-03-17T19:07:03.2223600Z * [new branch] gh/eellison/111/head -> origin/gh/eellison/111/head 2022-03-17T19:07:03.2224595Z * [new branch] gh/eellison/111/orig -> origin/gh/eellison/111/orig 2022-03-17T19:07:03.2226637Z * [new branch] gh/eellison/117/base -> origin/gh/eellison/117/base 2022-03-17T19:07:03.2227557Z * [new branch] gh/eellison/117/head -> origin/gh/eellison/117/head 2022-03-17T19:07:03.2229174Z * [new branch] gh/eellison/117/orig -> origin/gh/eellison/117/orig 2022-03-17T19:07:03.2231314Z * [new branch] gh/eellison/118/base -> origin/gh/eellison/118/base 2022-03-17T19:07:03.2236225Z * [new branch] gh/eellison/118/head -> origin/gh/eellison/118/head 2022-03-17T19:07:03.2236778Z * [new branch] gh/eellison/118/orig -> origin/gh/eellison/118/orig 2022-03-17T19:07:03.2237297Z * [new branch] gh/eellison/119/base -> origin/gh/eellison/119/base 2022-03-17T19:07:03.2237721Z * [new branch] gh/eellison/119/head -> origin/gh/eellison/119/head 2022-03-17T19:07:03.2238042Z * [new branch] gh/eellison/119/orig -> origin/gh/eellison/119/orig 2022-03-17T19:07:03.2239397Z * [new branch] gh/eellison/129/base -> origin/gh/eellison/129/base 2022-03-17T19:07:03.2240731Z * [new branch] gh/eellison/129/head -> origin/gh/eellison/129/head 2022-03-17T19:07:03.2242154Z * [new branch] gh/eellison/129/orig -> origin/gh/eellison/129/orig 2022-03-17T19:07:03.2244292Z * [new branch] gh/eellison/139/base -> origin/gh/eellison/139/base 2022-03-17T19:07:03.2245399Z * [new branch] gh/eellison/139/head -> origin/gh/eellison/139/head 2022-03-17T19:07:03.2246706Z * [new branch] gh/eellison/139/orig -> origin/gh/eellison/139/orig 2022-03-17T19:07:03.2248779Z * [new branch] gh/eellison/142/base -> origin/gh/eellison/142/base 2022-03-17T19:07:03.2250024Z * [new branch] gh/eellison/142/head -> origin/gh/eellison/142/head 2022-03-17T19:07:03.2251463Z * [new branch] gh/eellison/142/orig -> origin/gh/eellison/142/orig 2022-03-17T19:07:03.2253539Z * [new branch] gh/eellison/154/base -> origin/gh/eellison/154/base 2022-03-17T19:07:03.2254777Z * [new branch] gh/eellison/154/head -> origin/gh/eellison/154/head 2022-03-17T19:07:03.2256139Z * [new branch] gh/eellison/154/orig -> origin/gh/eellison/154/orig 2022-03-17T19:07:03.2258297Z * [new branch] gh/eellison/169/base -> origin/gh/eellison/169/base 2022-03-17T19:07:03.2259600Z * [new branch] gh/eellison/169/head -> origin/gh/eellison/169/head 2022-03-17T19:07:03.2261017Z * [new branch] gh/eellison/169/orig -> origin/gh/eellison/169/orig 2022-03-17T19:07:03.2263327Z * [new branch] gh/eellison/179/base -> origin/gh/eellison/179/base 2022-03-17T19:07:03.2264244Z * [new branch] gh/eellison/179/head -> origin/gh/eellison/179/head 2022-03-17T19:07:03.2265633Z * [new branch] gh/eellison/179/orig -> origin/gh/eellison/179/orig 2022-03-17T19:07:03.2267539Z * [new branch] gh/eellison/182/base -> origin/gh/eellison/182/base 2022-03-17T19:07:03.2268915Z * [new branch] gh/eellison/182/head -> origin/gh/eellison/182/head 2022-03-17T19:07:03.2270613Z * [new branch] gh/eellison/182/orig -> origin/gh/eellison/182/orig 2022-03-17T19:07:03.2272559Z * [new branch] gh/eellison/183/base -> origin/gh/eellison/183/base 2022-03-17T19:07:03.2273909Z * [new branch] gh/eellison/183/head -> origin/gh/eellison/183/head 2022-03-17T19:07:03.2275345Z * [new branch] gh/eellison/183/orig -> origin/gh/eellison/183/orig 2022-03-17T19:07:03.2277256Z * [new branch] gh/eellison/191/base -> origin/gh/eellison/191/base 2022-03-17T19:07:03.2278750Z * [new branch] gh/eellison/191/head -> origin/gh/eellison/191/head 2022-03-17T19:07:03.2280104Z * [new branch] gh/eellison/191/orig -> origin/gh/eellison/191/orig 2022-03-17T19:07:03.2282020Z * [new branch] gh/eellison/193/base -> origin/gh/eellison/193/base 2022-03-17T19:07:03.2283120Z * [new branch] gh/eellison/193/head -> origin/gh/eellison/193/head 2022-03-17T19:07:03.2284753Z * [new branch] gh/eellison/193/orig -> origin/gh/eellison/193/orig 2022-03-17T19:07:03.2286542Z * [new branch] gh/eellison/195/base -> origin/gh/eellison/195/base 2022-03-17T19:07:03.2287590Z * [new branch] gh/eellison/195/head -> origin/gh/eellison/195/head 2022-03-17T19:07:03.2289408Z * [new branch] gh/eellison/195/orig -> origin/gh/eellison/195/orig 2022-03-17T19:07:03.2290682Z * [new branch] gh/eellison/196/base -> origin/gh/eellison/196/base 2022-03-17T19:07:03.2292168Z * [new branch] gh/eellison/196/head -> origin/gh/eellison/196/head 2022-03-17T19:07:03.2293521Z * [new branch] gh/eellison/196/orig -> origin/gh/eellison/196/orig 2022-03-17T19:07:03.2295813Z * [new branch] gh/eellison/209/base -> origin/gh/eellison/209/base 2022-03-17T19:07:03.2297037Z * [new branch] gh/eellison/209/head -> origin/gh/eellison/209/head 2022-03-17T19:07:03.2298832Z * [new branch] gh/eellison/209/orig -> origin/gh/eellison/209/orig 2022-03-17T19:07:03.2300769Z * [new branch] gh/eellison/232/base -> origin/gh/eellison/232/base 2022-03-17T19:07:03.2302220Z * [new branch] gh/eellison/232/head -> origin/gh/eellison/232/head 2022-03-17T19:07:03.2303563Z * [new branch] gh/eellison/232/orig -> origin/gh/eellison/232/orig 2022-03-17T19:07:03.2305482Z * [new branch] gh/eellison/234/base -> origin/gh/eellison/234/base 2022-03-17T19:07:03.2306638Z * [new branch] gh/eellison/234/head -> origin/gh/eellison/234/head 2022-03-17T19:07:03.2308098Z * [new branch] gh/eellison/234/orig -> origin/gh/eellison/234/orig 2022-03-17T19:07:03.2310038Z * [new branch] gh/eellison/238/base -> origin/gh/eellison/238/base 2022-03-17T19:07:03.2311254Z * [new branch] gh/eellison/238/head -> origin/gh/eellison/238/head 2022-03-17T19:07:03.2313245Z * [new branch] gh/eellison/238/orig -> origin/gh/eellison/238/orig 2022-03-17T19:07:03.2314246Z * [new branch] gh/eellison/262/base -> origin/gh/eellison/262/base 2022-03-17T19:07:03.2315866Z * [new branch] gh/eellison/262/head -> origin/gh/eellison/262/head 2022-03-17T19:07:03.2316980Z * [new branch] gh/eellison/262/orig -> origin/gh/eellison/262/orig 2022-03-17T19:07:03.2319498Z * [new branch] gh/eellison/263/base -> origin/gh/eellison/263/base 2022-03-17T19:07:03.2320749Z * [new branch] gh/eellison/263/head -> origin/gh/eellison/263/head 2022-03-17T19:07:03.2322084Z * [new branch] gh/eellison/263/orig -> origin/gh/eellison/263/orig 2022-03-17T19:07:03.2323846Z * [new branch] gh/eellison/264/base -> origin/gh/eellison/264/base 2022-03-17T19:07:03.2324904Z * [new branch] gh/eellison/264/head -> origin/gh/eellison/264/head 2022-03-17T19:07:03.2326518Z * [new branch] gh/eellison/264/orig -> origin/gh/eellison/264/orig 2022-03-17T19:07:03.2327980Z * [new branch] gh/eellison/265/base -> origin/gh/eellison/265/base 2022-03-17T19:07:03.2329300Z * [new branch] gh/eellison/265/head -> origin/gh/eellison/265/head 2022-03-17T19:07:03.2330668Z * [new branch] gh/eellison/265/orig -> origin/gh/eellison/265/orig 2022-03-17T19:07:03.2332567Z * [new branch] gh/eellison/272/base -> origin/gh/eellison/272/base 2022-03-17T19:07:03.2333634Z * [new branch] gh/eellison/272/head -> origin/gh/eellison/272/head 2022-03-17T19:07:03.2335093Z * [new branch] gh/eellison/272/orig -> origin/gh/eellison/272/orig 2022-03-17T19:07:03.2337666Z * [new branch] gh/eellison/274/base -> origin/gh/eellison/274/base 2022-03-17T19:07:03.2338994Z * [new branch] gh/eellison/274/head -> origin/gh/eellison/274/head 2022-03-17T19:07:03.2340530Z * [new branch] gh/eellison/274/orig -> origin/gh/eellison/274/orig 2022-03-17T19:07:03.2342882Z * [new branch] gh/eellison/275/base -> origin/gh/eellison/275/base 2022-03-17T19:07:03.2344012Z * [new branch] gh/eellison/275/head -> origin/gh/eellison/275/head 2022-03-17T19:07:03.2345332Z * [new branch] gh/eellison/275/orig -> origin/gh/eellison/275/orig 2022-03-17T19:07:03.2347593Z * [new branch] gh/eellison/276/base -> origin/gh/eellison/276/base 2022-03-17T19:07:03.2348400Z * [new branch] gh/eellison/276/head -> origin/gh/eellison/276/head 2022-03-17T19:07:03.2349906Z * [new branch] gh/eellison/276/orig -> origin/gh/eellison/276/orig 2022-03-17T19:07:03.2351921Z * [new branch] gh/eellison/278/base -> origin/gh/eellison/278/base 2022-03-17T19:07:03.2353209Z * [new branch] gh/eellison/278/head -> origin/gh/eellison/278/head 2022-03-17T19:07:03.2354813Z * [new branch] gh/eellison/278/orig -> origin/gh/eellison/278/orig 2022-03-17T19:07:03.2357263Z * [new branch] gh/eellison/279/base -> origin/gh/eellison/279/base 2022-03-17T19:07:03.2359191Z * [new branch] gh/eellison/279/head -> origin/gh/eellison/279/head 2022-03-17T19:07:03.2360185Z * [new branch] gh/eellison/279/orig -> origin/gh/eellison/279/orig 2022-03-17T19:07:03.2362235Z * [new branch] gh/eellison/280/base -> origin/gh/eellison/280/base 2022-03-17T19:07:03.2364337Z * [new branch] gh/eellison/280/head -> origin/gh/eellison/280/head 2022-03-17T19:07:03.2366265Z * [new branch] gh/eellison/281/base -> origin/gh/eellison/281/base 2022-03-17T19:07:03.2367684Z * [new branch] gh/eellison/281/head -> origin/gh/eellison/281/head 2022-03-17T19:07:03.2369024Z * [new branch] gh/eellison/281/orig -> origin/gh/eellison/281/orig 2022-03-17T19:07:03.2371050Z * [new branch] gh/eellison/282/base -> origin/gh/eellison/282/base 2022-03-17T19:07:03.2372375Z * [new branch] gh/eellison/282/head -> origin/gh/eellison/282/head 2022-03-17T19:07:03.2373690Z * [new branch] gh/eellison/282/orig -> origin/gh/eellison/282/orig 2022-03-17T19:07:03.2375756Z * [new branch] gh/eellison/283/base -> origin/gh/eellison/283/base 2022-03-17T19:07:03.2377036Z * [new branch] gh/eellison/283/head -> origin/gh/eellison/283/head 2022-03-17T19:07:03.2378488Z * [new branch] gh/eellison/283/orig -> origin/gh/eellison/283/orig 2022-03-17T19:07:03.2380745Z * [new branch] gh/eellison/284/base -> origin/gh/eellison/284/base 2022-03-17T19:07:03.2382247Z * [new branch] gh/eellison/284/head -> origin/gh/eellison/284/head 2022-03-17T19:07:03.2383610Z * [new branch] gh/eellison/284/orig -> origin/gh/eellison/284/orig 2022-03-17T19:07:03.2385761Z * [new branch] gh/eellison/285/base -> origin/gh/eellison/285/base 2022-03-17T19:07:03.2386976Z * [new branch] gh/eellison/285/head -> origin/gh/eellison/285/head 2022-03-17T19:07:03.2389037Z * [new branch] gh/eellison/285/orig -> origin/gh/eellison/285/orig 2022-03-17T19:07:03.2390838Z * [new branch] gh/eellison/286/base -> origin/gh/eellison/286/base 2022-03-17T19:07:03.2392012Z * [new branch] gh/eellison/286/head -> origin/gh/eellison/286/head 2022-03-17T19:07:03.2393519Z * [new branch] gh/eellison/286/orig -> origin/gh/eellison/286/orig 2022-03-17T19:07:03.2395843Z * [new branch] gh/eellison/49/base -> origin/gh/eellison/49/base 2022-03-17T19:07:03.2396996Z * [new branch] gh/eellison/49/head -> origin/gh/eellison/49/head 2022-03-17T19:07:03.2398348Z * [new branch] gh/eellison/49/orig -> origin/gh/eellison/49/orig 2022-03-17T19:07:03.2400707Z * [new branch] gh/ejguan/24/base -> origin/gh/ejguan/24/base 2022-03-17T19:07:03.2402082Z * [new branch] gh/ejguan/24/head -> origin/gh/ejguan/24/head 2022-03-17T19:07:03.2403441Z * [new branch] gh/ejguan/24/orig -> origin/gh/ejguan/24/orig 2022-03-17T19:07:03.2404949Z * [new branch] gh/ejguan/25/base -> origin/gh/ejguan/25/base 2022-03-17T19:07:03.2407230Z * [new branch] gh/ejguan/25/head -> origin/gh/ejguan/25/head 2022-03-17T19:07:03.2408359Z * [new branch] gh/ejguan/25/orig -> origin/gh/ejguan/25/orig 2022-03-17T19:07:03.2410411Z * [new branch] gh/ejguan/26/base -> origin/gh/ejguan/26/base 2022-03-17T19:07:03.2411186Z * [new branch] gh/ejguan/26/head -> origin/gh/ejguan/26/head 2022-03-17T19:07:03.2412677Z * [new branch] gh/ejguan/26/orig -> origin/gh/ejguan/26/orig 2022-03-17T19:07:03.2414750Z * [new branch] gh/ejguan/27/base -> origin/gh/ejguan/27/base 2022-03-17T19:07:03.2416552Z * [new branch] gh/ejguan/27/head -> origin/gh/ejguan/27/head 2022-03-17T19:07:03.2417728Z * [new branch] gh/ejguan/27/orig -> origin/gh/ejguan/27/orig 2022-03-17T19:07:03.2419863Z * [new branch] gh/ejguan/3/base -> origin/gh/ejguan/3/base 2022-03-17T19:07:03.2421009Z * [new branch] gh/ejguan/3/head -> origin/gh/ejguan/3/head 2022-03-17T19:07:03.2422928Z * [new branch] gh/ejguan/3/orig -> origin/gh/ejguan/3/orig 2022-03-17T19:07:03.2424439Z * [new branch] gh/ejguan/30/base -> origin/gh/ejguan/30/base 2022-03-17T19:07:03.2425683Z * [new branch] gh/ejguan/30/head -> origin/gh/ejguan/30/head 2022-03-17T19:07:03.2427796Z * [new branch] gh/ejguan/33/base -> origin/gh/ejguan/33/base 2022-03-17T19:07:03.2428729Z * [new branch] gh/ejguan/33/head -> origin/gh/ejguan/33/head 2022-03-17T19:07:03.2430776Z * [new branch] gh/ejguan/40/base -> origin/gh/ejguan/40/base 2022-03-17T19:07:03.2432428Z * [new branch] gh/ejguan/40/head -> origin/gh/ejguan/40/head 2022-03-17T19:07:03.2433600Z * [new branch] gh/ejguan/40/orig -> origin/gh/ejguan/40/orig 2022-03-17T19:07:03.2435631Z * [new branch] gh/ejguan/68/base -> origin/gh/ejguan/68/base 2022-03-17T19:07:03.2436943Z * [new branch] gh/ejguan/68/head -> origin/gh/ejguan/68/head 2022-03-17T19:07:03.2438401Z * [new branch] gh/ejguan/68/orig -> origin/gh/ejguan/68/orig 2022-03-17T19:07:03.2440617Z * [new branch] gh/ejguan/71/base -> origin/gh/ejguan/71/base 2022-03-17T19:07:03.2441841Z * [new branch] gh/ejguan/71/head -> origin/gh/ejguan/71/head 2022-03-17T19:07:03.2443304Z * [new branch] gh/ejguan/71/orig -> origin/gh/ejguan/71/orig 2022-03-17T19:07:03.2444947Z * [new branch] gh/ejguan/72/base -> origin/gh/ejguan/72/base 2022-03-17T19:07:03.2446312Z * [new branch] gh/ejguan/72/head -> origin/gh/ejguan/72/head 2022-03-17T19:07:03.2447675Z * [new branch] gh/ejguan/72/orig -> origin/gh/ejguan/72/orig 2022-03-17T19:07:03.2449649Z * [new branch] gh/ejguan/73/base -> origin/gh/ejguan/73/base 2022-03-17T19:07:03.2450733Z * [new branch] gh/ejguan/73/head -> origin/gh/ejguan/73/head 2022-03-17T19:07:03.2452536Z * [new branch] gh/ejguan/73/orig -> origin/gh/ejguan/73/orig 2022-03-17T19:07:03.2454108Z * [new branch] gh/ejguan/74/base -> origin/gh/ejguan/74/base 2022-03-17T19:07:03.2455341Z * [new branch] gh/ejguan/74/head -> origin/gh/ejguan/74/head 2022-03-17T19:07:03.2456947Z * [new branch] gh/ejguan/74/orig -> origin/gh/ejguan/74/orig 2022-03-17T19:07:03.2458956Z * [new branch] gh/ejguan/89/base -> origin/gh/ejguan/89/base 2022-03-17T19:07:03.2460242Z * [new branch] gh/ejguan/89/head -> origin/gh/ejguan/89/head 2022-03-17T19:07:03.2461798Z * [new branch] gh/ejguan/89/orig -> origin/gh/ejguan/89/orig 2022-03-17T19:07:03.2463996Z * [new branch] gh/erjia-guan/1/base -> origin/gh/erjia-guan/1/base 2022-03-17T19:07:03.2465363Z * [new branch] gh/erjia-guan/1/head -> origin/gh/erjia-guan/1/head 2022-03-17T19:07:03.2466735Z * [new branch] gh/erjia-guan/1/orig -> origin/gh/erjia-guan/1/orig 2022-03-17T19:07:03.2468880Z * [new branch] gh/esqu1/1/base -> origin/gh/esqu1/1/base 2022-03-17T19:07:03.2470194Z * [new branch] gh/esqu1/1/head -> origin/gh/esqu1/1/head 2022-03-17T19:07:03.2472073Z * [new branch] gh/esqu1/1/orig -> origin/gh/esqu1/1/orig 2022-03-17T19:07:03.2473587Z * [new branch] gh/ethch18/1/base -> origin/gh/ethch18/1/base 2022-03-17T19:07:03.2475874Z * [new branch] gh/ezyang/1009/base -> origin/gh/ezyang/1009/base 2022-03-17T19:07:03.2477177Z * [new branch] gh/ezyang/1009/head -> origin/gh/ezyang/1009/head 2022-03-17T19:07:03.2478523Z * [new branch] gh/ezyang/1009/orig -> origin/gh/ezyang/1009/orig 2022-03-17T19:07:03.2480617Z * [new branch] gh/ezyang/1035/base -> origin/gh/ezyang/1035/base 2022-03-17T19:07:03.2481847Z * [new branch] gh/ezyang/1035/head -> origin/gh/ezyang/1035/head 2022-03-17T19:07:03.2483216Z * [new branch] gh/ezyang/1035/orig -> origin/gh/ezyang/1035/orig 2022-03-17T19:07:03.2485506Z * [new branch] gh/ezyang/1053/base -> origin/gh/ezyang/1053/base 2022-03-17T19:07:03.2487068Z * [new branch] gh/ezyang/1053/head -> origin/gh/ezyang/1053/head 2022-03-17T19:07:03.2488878Z * [new branch] gh/ezyang/1053/orig -> origin/gh/ezyang/1053/orig 2022-03-17T19:07:03.2490432Z * [new branch] gh/ezyang/1054/base -> origin/gh/ezyang/1054/base 2022-03-17T19:07:03.2491806Z * [new branch] gh/ezyang/1054/head -> origin/gh/ezyang/1054/head 2022-03-17T19:07:03.2493188Z * [new branch] gh/ezyang/1054/orig -> origin/gh/ezyang/1054/orig 2022-03-17T19:07:03.2520268Z * [new branch] gh/ezyang/1068/base -> origin/gh/ezyang/1068/base 2022-03-17T19:07:03.2520897Z * [new branch] gh/ezyang/1068/head -> origin/gh/ezyang/1068/head 2022-03-17T19:07:03.2521326Z * [new branch] gh/ezyang/1068/orig -> origin/gh/ezyang/1068/orig 2022-03-17T19:07:03.2521594Z * [new branch] gh/ezyang/1070/base -> origin/gh/ezyang/1070/base 2022-03-17T19:07:03.2521858Z * [new branch] gh/ezyang/1070/head -> origin/gh/ezyang/1070/head 2022-03-17T19:07:03.2522261Z * [new branch] gh/ezyang/1070/orig -> origin/gh/ezyang/1070/orig 2022-03-17T19:07:03.2522754Z * [new branch] gh/ezyang/1071/base -> origin/gh/ezyang/1071/base 2022-03-17T19:07:03.2523102Z * [new branch] gh/ezyang/1071/head -> origin/gh/ezyang/1071/head 2022-03-17T19:07:03.2523367Z * [new branch] gh/ezyang/1071/orig -> origin/gh/ezyang/1071/orig 2022-03-17T19:07:03.2523638Z * [new branch] gh/ezyang/1073/base -> origin/gh/ezyang/1073/base 2022-03-17T19:07:03.2524096Z * [new branch] gh/ezyang/1073/head -> origin/gh/ezyang/1073/head 2022-03-17T19:07:03.2524634Z * [new branch] gh/ezyang/1073/orig -> origin/gh/ezyang/1073/orig 2022-03-17T19:07:03.2524912Z * [new branch] gh/ezyang/1075/base -> origin/gh/ezyang/1075/base 2022-03-17T19:07:03.2525174Z * [new branch] gh/ezyang/1075/head -> origin/gh/ezyang/1075/head 2022-03-17T19:07:03.2525450Z * [new branch] gh/ezyang/1075/orig -> origin/gh/ezyang/1075/orig 2022-03-17T19:07:03.2525952Z * [new branch] gh/ezyang/1081/base -> origin/gh/ezyang/1081/base 2022-03-17T19:07:03.2526407Z * [new branch] gh/ezyang/1081/head -> origin/gh/ezyang/1081/head 2022-03-17T19:07:03.2526659Z * [new branch] gh/ezyang/1081/orig -> origin/gh/ezyang/1081/orig 2022-03-17T19:07:03.2527062Z * [new branch] gh/ezyang/1086/base -> origin/gh/ezyang/1086/base 2022-03-17T19:07:03.2527365Z * [new branch] gh/ezyang/1086/head -> origin/gh/ezyang/1086/head 2022-03-17T19:07:03.2527775Z * [new branch] gh/ezyang/1086/orig -> origin/gh/ezyang/1086/orig 2022-03-17T19:07:03.2528721Z * [new branch] gh/ezyang/1090/base -> origin/gh/ezyang/1090/base 2022-03-17T19:07:03.2530645Z * [new branch] gh/ezyang/1090/head -> origin/gh/ezyang/1090/head 2022-03-17T19:07:03.2532014Z * [new branch] gh/ezyang/1090/orig -> origin/gh/ezyang/1090/orig 2022-03-17T19:07:03.2533951Z * [new branch] gh/ezyang/1093/base -> origin/gh/ezyang/1093/base 2022-03-17T19:07:03.2535365Z * [new branch] gh/ezyang/1093/head -> origin/gh/ezyang/1093/head 2022-03-17T19:07:03.2537256Z * [new branch] gh/ezyang/1093/orig -> origin/gh/ezyang/1093/orig 2022-03-17T19:07:03.2539161Z * [new branch] gh/ezyang/1096/base -> origin/gh/ezyang/1096/base 2022-03-17T19:07:03.2540629Z * [new branch] gh/ezyang/1096/head -> origin/gh/ezyang/1096/head 2022-03-17T19:07:03.2541914Z * [new branch] gh/ezyang/1096/orig -> origin/gh/ezyang/1096/orig 2022-03-17T19:07:03.2543799Z * [new branch] gh/ezyang/1097/base -> origin/gh/ezyang/1097/base 2022-03-17T19:07:03.2545053Z * [new branch] gh/ezyang/1097/head -> origin/gh/ezyang/1097/head 2022-03-17T19:07:03.2546947Z * [new branch] gh/ezyang/1097/orig -> origin/gh/ezyang/1097/orig 2022-03-17T19:07:03.2548252Z * [new branch] gh/ezyang/1098/base -> origin/gh/ezyang/1098/base 2022-03-17T19:07:03.2549709Z * [new branch] gh/ezyang/1098/head -> origin/gh/ezyang/1098/head 2022-03-17T19:07:03.2551076Z * [new branch] gh/ezyang/1098/orig -> origin/gh/ezyang/1098/orig 2022-03-17T19:07:03.2553055Z * [new branch] gh/ezyang/1099/base -> origin/gh/ezyang/1099/base 2022-03-17T19:07:03.2554412Z * [new branch] gh/ezyang/1099/head -> origin/gh/ezyang/1099/head 2022-03-17T19:07:03.2555721Z * [new branch] gh/ezyang/1099/orig -> origin/gh/ezyang/1099/orig 2022-03-17T19:07:03.2557773Z * [new branch] gh/ezyang/1100/base -> origin/gh/ezyang/1100/base 2022-03-17T19:07:03.2559065Z * [new branch] gh/ezyang/1100/head -> origin/gh/ezyang/1100/head 2022-03-17T19:07:03.2560430Z * [new branch] gh/ezyang/1100/orig -> origin/gh/ezyang/1100/orig 2022-03-17T19:07:03.2562509Z * [new branch] gh/ezyang/1101/base -> origin/gh/ezyang/1101/base 2022-03-17T19:07:03.2563724Z * [new branch] gh/ezyang/1101/head -> origin/gh/ezyang/1101/head 2022-03-17T19:07:03.2565102Z * [new branch] gh/ezyang/1101/orig -> origin/gh/ezyang/1101/orig 2022-03-17T19:07:03.2567038Z * [new branch] gh/ezyang/1102/base -> origin/gh/ezyang/1102/base 2022-03-17T19:07:03.2568422Z * [new branch] gh/ezyang/1102/head -> origin/gh/ezyang/1102/head 2022-03-17T19:07:03.2569757Z * [new branch] gh/ezyang/1102/orig -> origin/gh/ezyang/1102/orig 2022-03-17T19:07:03.2572255Z * [new branch] gh/ezyang/1103/base -> origin/gh/ezyang/1103/base 2022-03-17T19:07:03.2573382Z * [new branch] gh/ezyang/1103/head -> origin/gh/ezyang/1103/head 2022-03-17T19:07:03.2575004Z * [new branch] gh/ezyang/1103/orig -> origin/gh/ezyang/1103/orig 2022-03-17T19:07:03.2576970Z * [new branch] gh/ezyang/1104/base -> origin/gh/ezyang/1104/base 2022-03-17T19:07:03.2578379Z * [new branch] gh/ezyang/1104/head -> origin/gh/ezyang/1104/head 2022-03-17T19:07:03.2579779Z * [new branch] gh/ezyang/1104/orig -> origin/gh/ezyang/1104/orig 2022-03-17T19:07:03.2582120Z * [new branch] gh/ezyang/1105/base -> origin/gh/ezyang/1105/base 2022-03-17T19:07:03.2583366Z * [new branch] gh/ezyang/1105/head -> origin/gh/ezyang/1105/head 2022-03-17T19:07:03.2584761Z * [new branch] gh/ezyang/1106/base -> origin/gh/ezyang/1106/base 2022-03-17T19:07:03.2586010Z * [new branch] gh/ezyang/1106/head -> origin/gh/ezyang/1106/head 2022-03-17T19:07:03.2587304Z * [new branch] gh/ezyang/1106/orig -> origin/gh/ezyang/1106/orig 2022-03-17T19:07:03.2588798Z * [new branch] gh/ezyang/1107/base -> origin/gh/ezyang/1107/base 2022-03-17T19:07:03.2590196Z * [new branch] gh/ezyang/1107/head -> origin/gh/ezyang/1107/head 2022-03-17T19:07:03.2591590Z * [new branch] gh/ezyang/1107/orig -> origin/gh/ezyang/1107/orig 2022-03-17T19:07:03.2593577Z * [new branch] gh/ezyang/269/base -> origin/gh/ezyang/269/base 2022-03-17T19:07:03.2594845Z * [new branch] gh/ezyang/269/head -> origin/gh/ezyang/269/head 2022-03-17T19:07:03.2596739Z * [new branch] gh/ezyang/269/orig -> origin/gh/ezyang/269/orig 2022-03-17T19:07:03.2598686Z * [new branch] gh/ezyang/301/base -> origin/gh/ezyang/301/base 2022-03-17T19:07:03.2599595Z * [new branch] gh/ezyang/301/head -> origin/gh/ezyang/301/head 2022-03-17T19:07:03.2601197Z * [new branch] gh/ezyang/301/orig -> origin/gh/ezyang/301/orig 2022-03-17T19:07:03.2603683Z * [new branch] gh/ezyang/366/head -> origin/gh/ezyang/366/head 2022-03-17T19:07:03.2605147Z * [new branch] gh/ezyang/366/orig -> origin/gh/ezyang/366/orig 2022-03-17T19:07:03.2607174Z * [new branch] gh/ezyang/371/base -> origin/gh/ezyang/371/base 2022-03-17T19:07:03.2609810Z * [new branch] gh/ezyang/371/head -> origin/gh/ezyang/371/head 2022-03-17T19:07:03.2612088Z * [new branch] gh/ezyang/371/orig -> origin/gh/ezyang/371/orig 2022-03-17T19:07:03.2614431Z * [new branch] gh/ezyang/401/base -> origin/gh/ezyang/401/base 2022-03-17T19:07:03.2615762Z * [new branch] gh/ezyang/401/head -> origin/gh/ezyang/401/head 2022-03-17T19:07:03.2618442Z * [new branch] gh/ezyang/479/base -> origin/gh/ezyang/479/base 2022-03-17T19:07:03.2619890Z * [new branch] gh/ezyang/479/head -> origin/gh/ezyang/479/head 2022-03-17T19:07:03.2621704Z * [new branch] gh/ezyang/479/orig -> origin/gh/ezyang/479/orig 2022-03-17T19:07:03.2624530Z * [new branch] gh/ezyang/495/head -> origin/gh/ezyang/495/head 2022-03-17T19:07:03.2626015Z * [new branch] gh/ezyang/495/orig -> origin/gh/ezyang/495/orig 2022-03-17T19:07:03.2627977Z * [new branch] gh/ezyang/497/base -> origin/gh/ezyang/497/base 2022-03-17T19:07:03.2629345Z * [new branch] gh/ezyang/497/head -> origin/gh/ezyang/497/head 2022-03-17T19:07:03.2631269Z * [new branch] gh/ezyang/497/orig -> origin/gh/ezyang/497/orig 2022-03-17T19:07:03.2632450Z * [new branch] gh/ezyang/501/base -> origin/gh/ezyang/501/base 2022-03-17T19:07:03.2634053Z * [new branch] gh/ezyang/501/head -> origin/gh/ezyang/501/head 2022-03-17T19:07:03.2635390Z * [new branch] gh/ezyang/501/orig -> origin/gh/ezyang/501/orig 2022-03-17T19:07:03.2638114Z * [new branch] gh/ezyang/502/base -> origin/gh/ezyang/502/base 2022-03-17T19:07:03.2639476Z * [new branch] gh/ezyang/502/head -> origin/gh/ezyang/502/head 2022-03-17T19:07:03.2641093Z * [new branch] gh/ezyang/502/orig -> origin/gh/ezyang/502/orig 2022-03-17T19:07:03.2642939Z * [new branch] gh/ezyang/503/base -> origin/gh/ezyang/503/base 2022-03-17T19:07:03.2644454Z * [new branch] gh/ezyang/503/head -> origin/gh/ezyang/503/head 2022-03-17T19:07:03.2645855Z * [new branch] gh/ezyang/503/orig -> origin/gh/ezyang/503/orig 2022-03-17T19:07:03.2647530Z * [new branch] gh/ezyang/504/base -> origin/gh/ezyang/504/base 2022-03-17T19:07:03.2648787Z * [new branch] gh/ezyang/504/head -> origin/gh/ezyang/504/head 2022-03-17T19:07:03.2650591Z * [new branch] gh/ezyang/504/orig -> origin/gh/ezyang/504/orig 2022-03-17T19:07:03.2651790Z * [new branch] gh/ezyang/505/base -> origin/gh/ezyang/505/base 2022-03-17T19:07:03.2653465Z * [new branch] gh/ezyang/505/head -> origin/gh/ezyang/505/head 2022-03-17T19:07:03.2654645Z * [new branch] gh/ezyang/505/orig -> origin/gh/ezyang/505/orig 2022-03-17T19:07:03.2656241Z * [new branch] gh/ezyang/506/base -> origin/gh/ezyang/506/base 2022-03-17T19:07:03.2658204Z * [new branch] gh/ezyang/506/head -> origin/gh/ezyang/506/head 2022-03-17T19:07:03.2659747Z * [new branch] gh/ezyang/506/orig -> origin/gh/ezyang/506/orig 2022-03-17T19:07:03.2662079Z * [new branch] gh/ezyang/546/base -> origin/gh/ezyang/546/base 2022-03-17T19:07:03.2663377Z * [new branch] gh/ezyang/546/head -> origin/gh/ezyang/546/head 2022-03-17T19:07:03.2664743Z * [new branch] gh/ezyang/546/orig -> origin/gh/ezyang/546/orig 2022-03-17T19:07:03.2666590Z * [new branch] gh/ezyang/547/base -> origin/gh/ezyang/547/base 2022-03-17T19:07:03.2667539Z * [new branch] gh/ezyang/547/head -> origin/gh/ezyang/547/head 2022-03-17T19:07:03.2669138Z * [new branch] gh/ezyang/547/orig -> origin/gh/ezyang/547/orig 2022-03-17T19:07:03.2670729Z * [new branch] gh/ezyang/548/base -> origin/gh/ezyang/548/base 2022-03-17T19:07:03.2672070Z * [new branch] gh/ezyang/548/head -> origin/gh/ezyang/548/head 2022-03-17T19:07:03.2673651Z * [new branch] gh/ezyang/548/orig -> origin/gh/ezyang/548/orig 2022-03-17T19:07:03.2675090Z * [new branch] gh/ezyang/549/base -> origin/gh/ezyang/549/base 2022-03-17T19:07:03.2676415Z * [new branch] gh/ezyang/549/head -> origin/gh/ezyang/549/head 2022-03-17T19:07:03.2677801Z * [new branch] gh/ezyang/549/orig -> origin/gh/ezyang/549/orig 2022-03-17T19:07:03.2679618Z * [new branch] gh/ezyang/550/base -> origin/gh/ezyang/550/base 2022-03-17T19:07:03.2681801Z * [new branch] gh/ezyang/550/head -> origin/gh/ezyang/550/head 2022-03-17T19:07:03.2683145Z * [new branch] gh/ezyang/550/orig -> origin/gh/ezyang/550/orig 2022-03-17T19:07:03.2684998Z * [new branch] gh/ezyang/551/base -> origin/gh/ezyang/551/base 2022-03-17T19:07:03.2685916Z * [new branch] gh/ezyang/551/head -> origin/gh/ezyang/551/head 2022-03-17T19:07:03.2687701Z * [new branch] gh/ezyang/551/orig -> origin/gh/ezyang/551/orig 2022-03-17T19:07:03.2688998Z * [new branch] gh/ezyang/552/base -> origin/gh/ezyang/552/base 2022-03-17T19:07:03.2690367Z * [new branch] gh/ezyang/552/head -> origin/gh/ezyang/552/head 2022-03-17T19:07:03.2691773Z * [new branch] gh/ezyang/552/orig -> origin/gh/ezyang/552/orig 2022-03-17T19:07:03.2693588Z * [new branch] gh/ezyang/553/base -> origin/gh/ezyang/553/base 2022-03-17T19:07:03.2694941Z * [new branch] gh/ezyang/553/head -> origin/gh/ezyang/553/head 2022-03-17T19:07:03.2696209Z * [new branch] gh/ezyang/553/orig -> origin/gh/ezyang/553/orig 2022-03-17T19:07:03.2698096Z * [new branch] gh/ezyang/554/base -> origin/gh/ezyang/554/base 2022-03-17T19:07:03.2699189Z * [new branch] gh/ezyang/554/head -> origin/gh/ezyang/554/head 2022-03-17T19:07:03.2701114Z * [new branch] gh/ezyang/554/orig -> origin/gh/ezyang/554/orig 2022-03-17T19:07:03.2703174Z * [new branch] gh/ezyang/568/base -> origin/gh/ezyang/568/base 2022-03-17T19:07:03.2704402Z * [new branch] gh/ezyang/568/head -> origin/gh/ezyang/568/head 2022-03-17T19:07:03.2705861Z * [new branch] gh/ezyang/568/orig -> origin/gh/ezyang/568/orig 2022-03-17T19:07:03.2708018Z * [new branch] gh/ezyang/573/base -> origin/gh/ezyang/573/base 2022-03-17T19:07:03.2708969Z * [new branch] gh/ezyang/573/head -> origin/gh/ezyang/573/head 2022-03-17T19:07:03.2710624Z * [new branch] gh/ezyang/573/orig -> origin/gh/ezyang/573/orig 2022-03-17T19:07:03.2712452Z * [new branch] gh/ezyang/574/base -> origin/gh/ezyang/574/base 2022-03-17T19:07:03.2713687Z * [new branch] gh/ezyang/574/head -> origin/gh/ezyang/574/head 2022-03-17T19:07:03.2715124Z * [new branch] gh/ezyang/574/orig -> origin/gh/ezyang/574/orig 2022-03-17T19:07:03.2716886Z * [new branch] gh/ezyang/575/base -> origin/gh/ezyang/575/base 2022-03-17T19:07:03.2718004Z * [new branch] gh/ezyang/575/head -> origin/gh/ezyang/575/head 2022-03-17T19:07:03.2719881Z * [new branch] gh/ezyang/575/orig -> origin/gh/ezyang/575/orig 2022-03-17T19:07:03.2721228Z * [new branch] gh/ezyang/576/base -> origin/gh/ezyang/576/base 2022-03-17T19:07:03.2722662Z * [new branch] gh/ezyang/576/head -> origin/gh/ezyang/576/head 2022-03-17T19:07:03.2724314Z * [new branch] gh/ezyang/576/orig -> origin/gh/ezyang/576/orig 2022-03-17T19:07:03.2725853Z * [new branch] gh/ezyang/577/base -> origin/gh/ezyang/577/base 2022-03-17T19:07:03.2727244Z * [new branch] gh/ezyang/577/head -> origin/gh/ezyang/577/head 2022-03-17T19:07:03.2728646Z * [new branch] gh/ezyang/577/orig -> origin/gh/ezyang/577/orig 2022-03-17T19:07:03.2732019Z * [new branch] gh/ezyang/590/base -> origin/gh/ezyang/590/base 2022-03-17T19:07:03.2734043Z * [new branch] gh/ezyang/590/head -> origin/gh/ezyang/590/head 2022-03-17T19:07:03.2735320Z * [new branch] gh/ezyang/590/orig -> origin/gh/ezyang/590/orig 2022-03-17T19:07:03.2737699Z * [new branch] gh/ezyang/604/base -> origin/gh/ezyang/604/base 2022-03-17T19:07:03.2739157Z * [new branch] gh/ezyang/604/head -> origin/gh/ezyang/604/head 2022-03-17T19:07:03.2741096Z * [new branch] gh/ezyang/604/orig -> origin/gh/ezyang/604/orig 2022-03-17T19:07:03.2743197Z * [new branch] gh/ezyang/608/base -> origin/gh/ezyang/608/base 2022-03-17T19:07:03.2744603Z * [new branch] gh/ezyang/608/head -> origin/gh/ezyang/608/head 2022-03-17T19:07:03.2745954Z * [new branch] gh/ezyang/608/orig -> origin/gh/ezyang/608/orig 2022-03-17T19:07:03.2748051Z * [new branch] gh/ezyang/613/base -> origin/gh/ezyang/613/base 2022-03-17T19:07:03.2749274Z * [new branch] gh/ezyang/613/head -> origin/gh/ezyang/613/head 2022-03-17T19:07:03.2750848Z * [new branch] gh/ezyang/613/orig -> origin/gh/ezyang/613/orig 2022-03-17T19:07:03.2753251Z * [new branch] gh/ezyang/617/base -> origin/gh/ezyang/617/base 2022-03-17T19:07:03.2756479Z * [new branch] gh/ezyang/617/head -> origin/gh/ezyang/617/head 2022-03-17T19:07:03.2756854Z * [new branch] gh/ezyang/617/orig -> origin/gh/ezyang/617/orig 2022-03-17T19:07:03.2757131Z * [new branch] gh/ezyang/620/base -> origin/gh/ezyang/620/base 2022-03-17T19:07:03.2758516Z * [new branch] gh/ezyang/620/head -> origin/gh/ezyang/620/head 2022-03-17T19:07:03.2761154Z * [new branch] gh/ezyang/620/orig -> origin/gh/ezyang/620/orig 2022-03-17T19:07:03.2762769Z * [new branch] gh/ezyang/624/base -> origin/gh/ezyang/624/base 2022-03-17T19:07:03.2764716Z * [new branch] gh/ezyang/624/head -> origin/gh/ezyang/624/head 2022-03-17T19:07:03.2765943Z * [new branch] gh/ezyang/624/orig -> origin/gh/ezyang/624/orig 2022-03-17T19:07:03.2768053Z * [new branch] gh/ezyang/628/base -> origin/gh/ezyang/628/base 2022-03-17T19:07:03.2770088Z * [new branch] gh/ezyang/628/head -> origin/gh/ezyang/628/head 2022-03-17T19:07:03.2770952Z * [new branch] gh/ezyang/628/orig -> origin/gh/ezyang/628/orig 2022-03-17T19:07:03.2772769Z * [new branch] gh/ezyang/630/base -> origin/gh/ezyang/630/base 2022-03-17T19:07:03.2774118Z * [new branch] gh/ezyang/630/head -> origin/gh/ezyang/630/head 2022-03-17T19:07:03.2775556Z * [new branch] gh/ezyang/630/orig -> origin/gh/ezyang/630/orig 2022-03-17T19:07:03.2777036Z * [new branch] gh/ezyang/631/base -> origin/gh/ezyang/631/base 2022-03-17T19:07:03.2778595Z * [new branch] gh/ezyang/631/head -> origin/gh/ezyang/631/head 2022-03-17T19:07:03.2779993Z * [new branch] gh/ezyang/631/orig -> origin/gh/ezyang/631/orig 2022-03-17T19:07:03.2782543Z * [new branch] gh/ezyang/632/base -> origin/gh/ezyang/632/base 2022-03-17T19:07:03.2783830Z * [new branch] gh/ezyang/632/head -> origin/gh/ezyang/632/head 2022-03-17T19:07:03.2785685Z * [new branch] gh/ezyang/632/orig -> origin/gh/ezyang/632/orig 2022-03-17T19:07:03.2787026Z * [new branch] gh/ezyang/633/base -> origin/gh/ezyang/633/base 2022-03-17T19:07:03.2788429Z * [new branch] gh/ezyang/633/head -> origin/gh/ezyang/633/head 2022-03-17T19:07:03.2789944Z * [new branch] gh/ezyang/633/orig -> origin/gh/ezyang/633/orig 2022-03-17T19:07:03.2792012Z * [new branch] gh/ezyang/635/base -> origin/gh/ezyang/635/base 2022-03-17T19:07:03.2793348Z * [new branch] gh/ezyang/635/head -> origin/gh/ezyang/635/head 2022-03-17T19:07:03.2794791Z * [new branch] gh/ezyang/635/orig -> origin/gh/ezyang/635/orig 2022-03-17T19:07:03.2796843Z * [new branch] gh/ezyang/636/base -> origin/gh/ezyang/636/base 2022-03-17T19:07:03.2798062Z * [new branch] gh/ezyang/636/head -> origin/gh/ezyang/636/head 2022-03-17T19:07:03.2799445Z * [new branch] gh/ezyang/636/orig -> origin/gh/ezyang/636/orig 2022-03-17T19:07:03.2801380Z * [new branch] gh/ezyang/637/base -> origin/gh/ezyang/637/base 2022-03-17T19:07:03.2802669Z * [new branch] gh/ezyang/637/head -> origin/gh/ezyang/637/head 2022-03-17T19:07:03.2804140Z * [new branch] gh/ezyang/637/orig -> origin/gh/ezyang/637/orig 2022-03-17T19:07:03.2806305Z * [new branch] gh/ezyang/638/base -> origin/gh/ezyang/638/base 2022-03-17T19:07:03.2807718Z * [new branch] gh/ezyang/638/head -> origin/gh/ezyang/638/head 2022-03-17T19:07:03.2809126Z * [new branch] gh/ezyang/638/orig -> origin/gh/ezyang/638/orig 2022-03-17T19:07:03.2810692Z * [new branch] gh/ezyang/640/base -> origin/gh/ezyang/640/base 2022-03-17T19:07:03.2812249Z * [new branch] gh/ezyang/640/head -> origin/gh/ezyang/640/head 2022-03-17T19:07:03.2813682Z * [new branch] gh/ezyang/640/orig -> origin/gh/ezyang/640/orig 2022-03-17T19:07:03.2815664Z * [new branch] gh/ezyang/641/base -> origin/gh/ezyang/641/base 2022-03-17T19:07:03.2817029Z * [new branch] gh/ezyang/641/head -> origin/gh/ezyang/641/head 2022-03-17T19:07:03.2818594Z * [new branch] gh/ezyang/641/orig -> origin/gh/ezyang/641/orig 2022-03-17T19:07:03.2821058Z * [new branch] gh/ezyang/644/base -> origin/gh/ezyang/644/base 2022-03-17T19:07:03.2823225Z * [new branch] gh/ezyang/644/head -> origin/gh/ezyang/644/head 2022-03-17T19:07:03.2824501Z * [new branch] gh/ezyang/644/orig -> origin/gh/ezyang/644/orig 2022-03-17T19:07:03.2826636Z * [new branch] gh/ezyang/648/base -> origin/gh/ezyang/648/base 2022-03-17T19:07:03.2827907Z * [new branch] gh/ezyang/648/head -> origin/gh/ezyang/648/head 2022-03-17T19:07:03.2829812Z * [new branch] gh/ezyang/648/orig -> origin/gh/ezyang/648/orig 2022-03-17T19:07:03.2831918Z * [new branch] gh/ezyang/650/base -> origin/gh/ezyang/650/base 2022-03-17T19:07:03.2833381Z * [new branch] gh/ezyang/650/head -> origin/gh/ezyang/650/head 2022-03-17T19:07:03.2835400Z * [new branch] gh/ezyang/650/orig -> origin/gh/ezyang/650/orig 2022-03-17T19:07:03.2836693Z * [new branch] gh/ezyang/671/base -> origin/gh/ezyang/671/base 2022-03-17T19:07:03.2838163Z * [new branch] gh/ezyang/671/head -> origin/gh/ezyang/671/head 2022-03-17T19:07:03.2839532Z * [new branch] gh/ezyang/671/orig -> origin/gh/ezyang/671/orig 2022-03-17T19:07:03.2841322Z * [new branch] gh/ezyang/672/base -> origin/gh/ezyang/672/base 2022-03-17T19:07:03.2842819Z * [new branch] gh/ezyang/672/head -> origin/gh/ezyang/672/head 2022-03-17T19:07:03.2844310Z * [new branch] gh/ezyang/672/orig -> origin/gh/ezyang/672/orig 2022-03-17T19:07:03.2846360Z * [new branch] gh/ezyang/754/base -> origin/gh/ezyang/754/base 2022-03-17T19:07:03.2847782Z * [new branch] gh/ezyang/754/head -> origin/gh/ezyang/754/head 2022-03-17T19:07:03.2850327Z * [new branch] gh/ezyang/780/head -> origin/gh/ezyang/780/head 2022-03-17T19:07:03.2852395Z * [new branch] gh/ezyang/811/base -> origin/gh/ezyang/811/base 2022-03-17T19:07:03.2853548Z * [new branch] gh/ezyang/811/orig -> origin/gh/ezyang/811/orig 2022-03-17T19:07:03.2855599Z * [new branch] gh/ezyang/838/base -> origin/gh/ezyang/838/base 2022-03-17T19:07:03.2856887Z * [new branch] gh/ezyang/838/head -> origin/gh/ezyang/838/head 2022-03-17T19:07:03.2859018Z * [new branch] gh/ezyang/838/orig -> origin/gh/ezyang/838/orig 2022-03-17T19:07:03.2861176Z * [new branch] gh/ezyang/870/base -> origin/gh/ezyang/870/base 2022-03-17T19:07:03.2863308Z * [new branch] gh/ezyang/870/head -> origin/gh/ezyang/870/head 2022-03-17T19:07:03.2864288Z * [new branch] gh/ezyang/870/orig -> origin/gh/ezyang/870/orig 2022-03-17T19:07:03.2866846Z * [new branch] gh/ezyang/906/base -> origin/gh/ezyang/906/base 2022-03-17T19:07:03.2868211Z * [new branch] gh/ezyang/906/head -> origin/gh/ezyang/906/head 2022-03-17T19:07:03.2870034Z * [new branch] gh/ezyang/906/orig -> origin/gh/ezyang/906/orig 2022-03-17T19:07:03.2871901Z * [new branch] gh/ezyang/915/base -> origin/gh/ezyang/915/base 2022-03-17T19:07:03.2873199Z * [new branch] gh/ezyang/915/head -> origin/gh/ezyang/915/head 2022-03-17T19:07:03.2874900Z * [new branch] gh/ezyang/915/orig -> origin/gh/ezyang/915/orig 2022-03-17T19:07:03.2876713Z * [new branch] gh/ezyang/932/base -> origin/gh/ezyang/932/base 2022-03-17T19:07:03.2877777Z * [new branch] gh/ezyang/932/head -> origin/gh/ezyang/932/head 2022-03-17T19:07:03.2879370Z * [new branch] gh/ezyang/932/orig -> origin/gh/ezyang/932/orig 2022-03-17T19:07:03.2881469Z * [new branch] gh/ezyang/933/base -> origin/gh/ezyang/933/base 2022-03-17T19:07:03.2882616Z * [new branch] gh/ezyang/933/head -> origin/gh/ezyang/933/head 2022-03-17T19:07:03.2883988Z * [new branch] gh/ezyang/933/orig -> origin/gh/ezyang/933/orig 2022-03-17T19:07:03.2886457Z * [new branch] gh/ezyang/952/base -> origin/gh/ezyang/952/base 2022-03-17T19:07:03.2888420Z * [new branch] gh/ezyang/952/head -> origin/gh/ezyang/952/head 2022-03-17T19:07:03.2889306Z * [new branch] gh/ezyang/952/orig -> origin/gh/ezyang/952/orig 2022-03-17T19:07:03.2891560Z * [new branch] gh/ezyang/993/base -> origin/gh/ezyang/993/base 2022-03-17T19:07:03.2893759Z * [new branch] gh/ezyang/993/head -> origin/gh/ezyang/993/head 2022-03-17T19:07:03.2895007Z * [new branch] gh/ezyang/993/orig -> origin/gh/ezyang/993/orig 2022-03-17T19:07:03.2897263Z * [new branch] gh/ezyang/994/base -> origin/gh/ezyang/994/base 2022-03-17T19:07:03.2898291Z * [new branch] gh/ezyang/994/head -> origin/gh/ezyang/994/head 2022-03-17T19:07:03.2899789Z * [new branch] gh/ezyang/994/orig -> origin/gh/ezyang/994/orig 2022-03-17T19:07:03.2901482Z * [new branch] gh/ezyang/995/base -> origin/gh/ezyang/995/base 2022-03-17T19:07:03.2903349Z * [new branch] gh/ezyang/995/head -> origin/gh/ezyang/995/head 2022-03-17T19:07:03.2904678Z * [new branch] gh/ezyang/995/orig -> origin/gh/ezyang/995/orig 2022-03-17T19:07:03.2906635Z * [new branch] gh/ezyang/997/base -> origin/gh/ezyang/997/base 2022-03-17T19:07:03.2907890Z * [new branch] gh/ezyang/997/head -> origin/gh/ezyang/997/head 2022-03-17T19:07:03.2909343Z * [new branch] gh/ezyang/997/orig -> origin/gh/ezyang/997/orig 2022-03-17T19:07:03.2911740Z * [new branch] gh/fduwjj/23/base -> origin/gh/fduwjj/23/base 2022-03-17T19:07:03.2913088Z * [new branch] gh/fduwjj/23/head -> origin/gh/fduwjj/23/head 2022-03-17T19:07:03.2914842Z * [new branch] gh/fduwjj/23/orig -> origin/gh/fduwjj/23/orig 2022-03-17T19:07:03.2916775Z * [new branch] gh/fegin/4/base -> origin/gh/fegin/4/base 2022-03-17T19:07:03.2918068Z * [new branch] gh/fegin/4/head -> origin/gh/fegin/4/head 2022-03-17T19:07:03.2919517Z * [new branch] gh/fegin/4/orig -> origin/gh/fegin/4/orig 2022-03-17T19:07:03.2921689Z * [new branch] gh/firstprayer/1/base -> origin/gh/firstprayer/1/base 2022-03-17T19:07:03.2923039Z * [new branch] gh/firstprayer/1/head -> origin/gh/firstprayer/1/head 2022-03-17T19:07:03.2924535Z * [new branch] gh/firstprayer/1/orig -> origin/gh/firstprayer/1/orig 2022-03-17T19:07:03.2926520Z * [new branch] gh/gamrix/13/base -> origin/gh/gamrix/13/base 2022-03-17T19:07:03.2928134Z * [new branch] gh/gamrix/13/orig -> origin/gh/gamrix/13/orig 2022-03-17T19:07:03.2930051Z * [new branch] gh/gamrix/14/base -> origin/gh/gamrix/14/base 2022-03-17T19:07:03.2931121Z * [new branch] gh/gamrix/14/orig -> origin/gh/gamrix/14/orig 2022-03-17T19:07:03.2933078Z * [new branch] gh/gamrix/15/base -> origin/gh/gamrix/15/base 2022-03-17T19:07:03.2934311Z * [new branch] gh/gamrix/15/orig -> origin/gh/gamrix/15/orig 2022-03-17T19:07:03.2936213Z * [new branch] gh/gamrix/16/base -> origin/gh/gamrix/16/base 2022-03-17T19:07:03.2937480Z * [new branch] gh/gamrix/16/orig -> origin/gh/gamrix/16/orig 2022-03-17T19:07:03.2939490Z * [new branch] gh/gamrix/17/base -> origin/gh/gamrix/17/base 2022-03-17T19:07:03.2940712Z * [new branch] gh/gamrix/17/head -> origin/gh/gamrix/17/head 2022-03-17T19:07:03.2942746Z * [new branch] gh/gamrix/17/orig -> origin/gh/gamrix/17/orig 2022-03-17T19:07:03.2943880Z * [new branch] gh/gamrix/22/base -> origin/gh/gamrix/22/base 2022-03-17T19:07:03.2945293Z * [new branch] gh/gamrix/22/orig -> origin/gh/gamrix/22/orig 2022-03-17T19:07:03.2947448Z * [new branch] gh/gamrix/23/base -> origin/gh/gamrix/23/base 2022-03-17T19:07:03.2948577Z * [new branch] gh/gamrix/23/orig -> origin/gh/gamrix/23/orig 2022-03-17T19:07:03.2950547Z * [new branch] gh/gamrix/25/base -> origin/gh/gamrix/25/base 2022-03-17T19:07:03.2951819Z * [new branch] gh/gamrix/25/head -> origin/gh/gamrix/25/head 2022-03-17T19:07:03.2953224Z * [new branch] gh/gamrix/25/orig -> origin/gh/gamrix/25/orig 2022-03-17T19:07:03.2955371Z * [new branch] gh/gamrix/26/base -> origin/gh/gamrix/26/base 2022-03-17T19:07:03.2956366Z * [new branch] gh/gamrix/26/head -> origin/gh/gamrix/26/head 2022-03-17T19:07:03.2957792Z * [new branch] gh/gamrix/26/orig -> origin/gh/gamrix/26/orig 2022-03-17T19:07:03.2959837Z * [new branch] gh/gamrix/30/base -> origin/gh/gamrix/30/base 2022-03-17T19:07:03.2960783Z * [new branch] gh/gamrix/30/head -> origin/gh/gamrix/30/head 2022-03-17T19:07:03.2962595Z * [new branch] gh/gamrix/30/orig -> origin/gh/gamrix/30/orig 2022-03-17T19:07:03.2963997Z * [new branch] gh/gamrix/43/base -> origin/gh/gamrix/43/base 2022-03-17T19:07:03.2965874Z * [new branch] gh/gamrix/43/head -> origin/gh/gamrix/43/head 2022-03-17T19:07:03.2966840Z * [new branch] gh/gamrix/43/orig -> origin/gh/gamrix/43/orig 2022-03-17T19:07:03.2969023Z * [new branch] gh/gamrix/44/base -> origin/gh/gamrix/44/base 2022-03-17T19:07:03.2970201Z * [new branch] gh/gamrix/44/head -> origin/gh/gamrix/44/head 2022-03-17T19:07:03.2971339Z * [new branch] gh/gamrix/44/orig -> origin/gh/gamrix/44/orig 2022-03-17T19:07:03.2972898Z * [new branch] gh/gamrix/46/base -> origin/gh/gamrix/46/base 2022-03-17T19:07:03.2974267Z * [new branch] gh/gamrix/46/head -> origin/gh/gamrix/46/head 2022-03-17T19:07:03.2975601Z * [new branch] gh/gamrix/46/orig -> origin/gh/gamrix/46/orig 2022-03-17T19:07:03.2977060Z * [new branch] gh/gamrix/47/base -> origin/gh/gamrix/47/base 2022-03-17T19:07:03.2978451Z * [new branch] gh/gamrix/47/head -> origin/gh/gamrix/47/head 2022-03-17T19:07:03.2979838Z * [new branch] gh/gamrix/47/orig -> origin/gh/gamrix/47/orig 2022-03-17T19:07:03.2982193Z * [new branch] gh/gamrix/49/base -> origin/gh/gamrix/49/base 2022-03-17T19:07:03.2983376Z * [new branch] gh/gamrix/49/head -> origin/gh/gamrix/49/head 2022-03-17T19:07:03.2985326Z * [new branch] gh/gamrix/49/orig -> origin/gh/gamrix/49/orig 2022-03-17T19:07:03.2986770Z * [new branch] gh/gamrix/50/base -> origin/gh/gamrix/50/base 2022-03-17T19:07:03.2988153Z * [new branch] gh/gamrix/50/head -> origin/gh/gamrix/50/head 2022-03-17T19:07:03.2989527Z * [new branch] gh/gamrix/50/orig -> origin/gh/gamrix/50/orig 2022-03-17T19:07:03.2991581Z * [new branch] gh/gamrix/51/base -> origin/gh/gamrix/51/base 2022-03-17T19:07:03.2992747Z * [new branch] gh/gamrix/51/head -> origin/gh/gamrix/51/head 2022-03-17T19:07:03.2994081Z * [new branch] gh/gamrix/51/orig -> origin/gh/gamrix/51/orig 2022-03-17T19:07:03.2996290Z * [new branch] gh/gchanan/110/head -> origin/gh/gchanan/110/head 2022-03-17T19:07:03.2998230Z * [new branch] gh/gchanan/166/base -> origin/gh/gchanan/166/base 2022-03-17T19:07:03.2999330Z * [new branch] gh/gchanan/166/head -> origin/gh/gchanan/166/head 2022-03-17T19:07:03.3000712Z * [new branch] gh/gchanan/166/orig -> origin/gh/gchanan/166/orig 2022-03-17T19:07:03.3002874Z * [new branch] gh/gchanan/178/base -> origin/gh/gchanan/178/base 2022-03-17T19:07:03.3004166Z * [new branch] gh/gchanan/178/head -> origin/gh/gchanan/178/head 2022-03-17T19:07:03.3005504Z * [new branch] gh/gchanan/178/orig -> origin/gh/gchanan/178/orig 2022-03-17T19:07:03.3007545Z * [new branch] gh/gchanan/268/base -> origin/gh/gchanan/268/base 2022-03-17T19:07:03.3008807Z * [new branch] gh/gchanan/268/head -> origin/gh/gchanan/268/head 2022-03-17T19:07:03.3010389Z * [new branch] gh/gchanan/268/orig -> origin/gh/gchanan/268/orig 2022-03-17T19:07:03.3012431Z * [new branch] gh/gchanan/269/base -> origin/gh/gchanan/269/base 2022-03-17T19:07:03.3013618Z * [new branch] gh/gchanan/269/head -> origin/gh/gchanan/269/head 2022-03-17T19:07:03.3015006Z * [new branch] gh/gchanan/269/orig -> origin/gh/gchanan/269/orig 2022-03-17T19:07:03.3016995Z * [new branch] gh/gchanan/302/base -> origin/gh/gchanan/302/base 2022-03-17T19:07:03.3018233Z * [new branch] gh/gchanan/302/head -> origin/gh/gchanan/302/head 2022-03-17T19:07:03.3020358Z * [new branch] gh/gchanan/302/orig -> origin/gh/gchanan/302/orig 2022-03-17T19:07:03.3023014Z * [new branch] gh/gchanan/329/base -> origin/gh/gchanan/329/base 2022-03-17T19:07:03.3024862Z * [new branch] gh/gchanan/329/head -> origin/gh/gchanan/329/head 2022-03-17T19:07:03.3026030Z * [new branch] gh/gchanan/329/orig -> origin/gh/gchanan/329/orig 2022-03-17T19:07:03.3028095Z * [new branch] gh/gchanan/330/base -> origin/gh/gchanan/330/base 2022-03-17T19:07:03.3029014Z * [new branch] gh/gchanan/330/head -> origin/gh/gchanan/330/head 2022-03-17T19:07:03.3030432Z * [new branch] gh/gchanan/330/orig -> origin/gh/gchanan/330/orig 2022-03-17T19:07:03.3032946Z * [new branch] gh/gchanan/336/base -> origin/gh/gchanan/336/base 2022-03-17T19:07:03.3034254Z * [new branch] gh/gchanan/336/head -> origin/gh/gchanan/336/head 2022-03-17T19:07:03.3036013Z * [new branch] gh/gchanan/336/orig -> origin/gh/gchanan/336/orig 2022-03-17T19:07:03.3037825Z * [new branch] gh/gchanan/45/base -> origin/gh/gchanan/45/base 2022-03-17T19:07:03.3039084Z * [new branch] gh/gchanan/45/head -> origin/gh/gchanan/45/head 2022-03-17T19:07:03.3040399Z * [new branch] gh/gchanan/45/orig -> origin/gh/gchanan/45/orig 2022-03-17T19:07:03.3042471Z * [new branch] gh/gchanan/64/base -> origin/gh/gchanan/64/base 2022-03-17T19:07:03.3043697Z * [new branch] gh/gchanan/64/head -> origin/gh/gchanan/64/head 2022-03-17T19:07:03.3045417Z * [new branch] gh/gchanan/64/orig -> origin/gh/gchanan/64/orig 2022-03-17T19:07:03.3046815Z * [new branch] gh/gchanan/65/base -> origin/gh/gchanan/65/base 2022-03-17T19:07:03.3048144Z * [new branch] gh/gchanan/65/head -> origin/gh/gchanan/65/head 2022-03-17T19:07:03.3049505Z * [new branch] gh/gchanan/65/orig -> origin/gh/gchanan/65/orig 2022-03-17T19:07:03.3051606Z * [new branch] gh/gchanan/70/base -> origin/gh/gchanan/70/base 2022-03-17T19:07:03.3052818Z * [new branch] gh/gchanan/70/head -> origin/gh/gchanan/70/head 2022-03-17T19:07:03.3054235Z * [new branch] gh/gchanan/70/orig -> origin/gh/gchanan/70/orig 2022-03-17T19:07:03.3055804Z * [new branch] gh/gchanan/71/base -> origin/gh/gchanan/71/base 2022-03-17T19:07:03.3057210Z * [new branch] gh/gchanan/71/head -> origin/gh/gchanan/71/head 2022-03-17T19:07:03.3058523Z * [new branch] gh/gchanan/71/orig -> origin/gh/gchanan/71/orig 2022-03-17T19:07:03.3060431Z * [new branch] gh/gchanan/72/base -> origin/gh/gchanan/72/base 2022-03-17T19:07:03.3061511Z * [new branch] gh/gchanan/72/head -> origin/gh/gchanan/72/head 2022-03-17T19:07:03.3063420Z * [new branch] gh/gchanan/72/orig -> origin/gh/gchanan/72/orig 2022-03-17T19:07:03.3065548Z * [new branch] gh/gchanan/80/base -> origin/gh/gchanan/80/base 2022-03-17T19:07:03.3066538Z * [new branch] gh/gchanan/80/head -> origin/gh/gchanan/80/head 2022-03-17T19:07:03.3067932Z * [new branch] gh/gchanan/80/orig -> origin/gh/gchanan/80/orig 2022-03-17T19:07:03.3070104Z * [new branch] gh/gcramer23/1/base -> origin/gh/gcramer23/1/base 2022-03-17T19:07:03.3071622Z * [new branch] gh/gcramer23/1/head -> origin/gh/gcramer23/1/head 2022-03-17T19:07:03.3072950Z * [new branch] gh/gcramer23/1/orig -> origin/gh/gcramer23/1/orig 2022-03-17T19:07:03.3075030Z * [new branch] gh/gcramer23/12/base -> origin/gh/gcramer23/12/base 2022-03-17T19:07:03.3076094Z * [new branch] gh/gcramer23/12/orig -> origin/gh/gcramer23/12/orig 2022-03-17T19:07:03.3078220Z * [new branch] gh/gcramer23/13/base -> origin/gh/gcramer23/13/base 2022-03-17T19:07:03.3079112Z * [new branch] gh/gcramer23/13/head -> origin/gh/gcramer23/13/head 2022-03-17T19:07:03.3081440Z * [new branch] gh/gcramer23/13/orig -> origin/gh/gcramer23/13/orig 2022-03-17T19:07:03.3083691Z * [new branch] gh/gcramer23/16/base -> origin/gh/gcramer23/16/base 2022-03-17T19:07:03.3084688Z * [new branch] gh/gcramer23/16/orig -> origin/gh/gcramer23/16/orig 2022-03-17T19:07:03.3087215Z * [new branch] gh/gcramer23/17/base -> origin/gh/gcramer23/17/base 2022-03-17T19:07:03.3088470Z * [new branch] gh/gcramer23/17/orig -> origin/gh/gcramer23/17/orig 2022-03-17T19:07:03.3090647Z * [new branch] gh/gcramer23/19/base -> origin/gh/gcramer23/19/base 2022-03-17T19:07:03.3091544Z * [new branch] gh/gcramer23/19/head -> origin/gh/gcramer23/19/head 2022-03-17T19:07:03.3093701Z * [new branch] gh/gcramer23/2/base -> origin/gh/gcramer23/2/base 2022-03-17T19:07:03.3095070Z * [new branch] gh/gcramer23/2/head -> origin/gh/gcramer23/2/head 2022-03-17T19:07:03.3096461Z * [new branch] gh/gcramer23/2/orig -> origin/gh/gcramer23/2/orig 2022-03-17T19:07:03.3098332Z * [new branch] gh/gcramer23/20/base -> origin/gh/gcramer23/20/base 2022-03-17T19:07:03.3099312Z * [new branch] gh/gcramer23/20/head -> origin/gh/gcramer23/20/head 2022-03-17T19:07:03.3101002Z * [new branch] gh/gcramer23/20/orig -> origin/gh/gcramer23/20/orig 2022-03-17T19:07:03.3103273Z * [new branch] gh/gcramer23/21/base -> origin/gh/gcramer23/21/base 2022-03-17T19:07:03.3104444Z * [new branch] gh/gcramer23/21/orig -> origin/gh/gcramer23/21/orig 2022-03-17T19:07:03.3106601Z * [new branch] gh/gcramer23/23/base -> origin/gh/gcramer23/23/base 2022-03-17T19:07:03.3107438Z * [new branch] gh/gcramer23/23/head -> origin/gh/gcramer23/23/head 2022-03-17T19:07:03.3108853Z * [new branch] gh/gcramer23/23/orig -> origin/gh/gcramer23/23/orig 2022-03-17T19:07:03.3110919Z * [new branch] gh/gcramer23/5/base -> origin/gh/gcramer23/5/base 2022-03-17T19:07:03.3112171Z * [new branch] gh/gcramer23/5/orig -> origin/gh/gcramer23/5/orig 2022-03-17T19:07:03.3113696Z * [new branch] gh/gcramer23/6/base -> origin/gh/gcramer23/6/base 2022-03-17T19:07:03.3114971Z * [new branch] gh/gcramer23/6/head -> origin/gh/gcramer23/6/head 2022-03-17T19:07:03.3125147Z * [new branch] gh/gcramer23/8/base -> origin/gh/gcramer23/8/base 2022-03-17T19:07:03.3126597Z * [new branch] gh/gcramer23/8/head -> origin/gh/gcramer23/8/head 2022-03-17T19:07:03.3127886Z * [new branch] gh/gcramer23/8/orig -> origin/gh/gcramer23/8/orig 2022-03-17T19:07:03.3130295Z * [new branch] gh/george-qi/19/base -> origin/gh/george-qi/19/base 2022-03-17T19:07:03.3131622Z * [new branch] gh/george-qi/19/head -> origin/gh/george-qi/19/head 2022-03-17T19:07:03.3132992Z * [new branch] gh/george-qi/19/orig -> origin/gh/george-qi/19/orig 2022-03-17T19:07:03.3135138Z * [new branch] gh/george-qi/20/base -> origin/gh/george-qi/20/base 2022-03-17T19:07:03.3135958Z * [new branch] gh/george-qi/20/head -> origin/gh/george-qi/20/head 2022-03-17T19:07:03.3137606Z * [new branch] gh/george-qi/20/orig -> origin/gh/george-qi/20/orig 2022-03-17T19:07:03.3139493Z * [new branch] gh/george-qi/21/base -> origin/gh/george-qi/21/base 2022-03-17T19:07:03.3141210Z * [new branch] gh/george-qi/21/head -> origin/gh/george-qi/21/head 2022-03-17T19:07:03.3143210Z * [new branch] gh/george-qi/21/orig -> origin/gh/george-qi/21/orig 2022-03-17T19:07:03.3144567Z * [new branch] gh/george-qi/22/base -> origin/gh/george-qi/22/base 2022-03-17T19:07:03.3150009Z * [new branch] gh/george-qi/22/head -> origin/gh/george-qi/22/head 2022-03-17T19:07:03.3151268Z * [new branch] gh/george-qi/22/orig -> origin/gh/george-qi/22/orig 2022-03-17T19:07:03.3153230Z * [new branch] gh/george-qi/23/base -> origin/gh/george-qi/23/base 2022-03-17T19:07:03.3154237Z * [new branch] gh/george-qi/23/head -> origin/gh/george-qi/23/head 2022-03-17T19:07:03.3155698Z * [new branch] gh/george-qi/23/orig -> origin/gh/george-qi/23/orig 2022-03-17T19:07:03.3157219Z * [new branch] gh/george-qi/24/base -> origin/gh/george-qi/24/base 2022-03-17T19:07:03.3160573Z * [new branch] gh/george-qi/24/head -> origin/gh/george-qi/24/head 2022-03-17T19:07:03.3163321Z * [new branch] gh/george-qi/24/orig -> origin/gh/george-qi/24/orig 2022-03-17T19:07:03.3166587Z * [new branch] gh/george-qi/25/base -> origin/gh/george-qi/25/base 2022-03-17T19:07:03.3167984Z * [new branch] gh/george-qi/25/head -> origin/gh/george-qi/25/head 2022-03-17T19:07:03.3170193Z * [new branch] gh/george-qi/25/orig -> origin/gh/george-qi/25/orig 2022-03-17T19:07:03.3172292Z * [new branch] gh/glaringlee/31/base -> origin/gh/glaringlee/31/base 2022-03-17T19:07:03.3174243Z * [new branch] gh/glaringlee/31/head -> origin/gh/glaringlee/31/head 2022-03-17T19:07:03.3175541Z * [new branch] gh/glaringlee/31/orig -> origin/gh/glaringlee/31/orig 2022-03-17T19:07:03.3180522Z * [new branch] gh/gqchen/1/base -> origin/gh/gqchen/1/base 2022-03-17T19:07:03.3182272Z * [new branch] gh/gqchen/1/head -> origin/gh/gqchen/1/head 2022-03-17T19:07:03.3183702Z * [new branch] gh/gqchen/1/orig -> origin/gh/gqchen/1/orig 2022-03-17T19:07:03.3188917Z * [new branch] gh/haichuan-fb/2/base -> origin/gh/haichuan-fb/2/base 2022-03-17T19:07:03.3190273Z * [new branch] gh/haichuan-fb/2/orig -> origin/gh/haichuan-fb/2/orig 2022-03-17T19:07:03.3192493Z * [new branch] gh/hameerabbasi/5/base -> origin/gh/hameerabbasi/5/base 2022-03-17T19:07:03.3195188Z * [new branch] gh/hameerabbasi/5/orig -> origin/gh/hameerabbasi/5/orig 2022-03-17T19:07:03.3197654Z * [new branch] gh/heitorschueroff/14/base -> origin/gh/heitorschueroff/14/base 2022-03-17T19:07:03.3199154Z * [new branch] gh/heitorschueroff/14/head -> origin/gh/heitorschueroff/14/head 2022-03-17T19:07:03.3200569Z * [new branch] gh/heitorschueroff/14/orig -> origin/gh/heitorschueroff/14/orig 2022-03-17T19:07:03.3202725Z * [new branch] gh/heitorschueroff/17/base -> origin/gh/heitorschueroff/17/base 2022-03-17T19:07:03.3203912Z * [new branch] gh/heitorschueroff/17/head -> origin/gh/heitorschueroff/17/head 2022-03-17T19:07:03.3205307Z * [new branch] gh/heitorschueroff/17/orig -> origin/gh/heitorschueroff/17/orig 2022-03-17T19:07:03.3207359Z * [new branch] gh/heitorschueroff/18/base -> origin/gh/heitorschueroff/18/base 2022-03-17T19:07:03.3208435Z * [new branch] gh/heitorschueroff/18/head -> origin/gh/heitorschueroff/18/head 2022-03-17T19:07:03.3209838Z * [new branch] gh/heitorschueroff/18/orig -> origin/gh/heitorschueroff/18/orig 2022-03-17T19:07:03.3211945Z * [new branch] gh/heitorschueroff/19/base -> origin/gh/heitorschueroff/19/base 2022-03-17T19:07:03.3212833Z * [new branch] gh/heitorschueroff/19/head -> origin/gh/heitorschueroff/19/head 2022-03-17T19:07:03.3214596Z * [new branch] gh/heitorschueroff/19/orig -> origin/gh/heitorschueroff/19/orig 2022-03-17T19:07:03.3216949Z * [new branch] gh/houseroad/9/base -> origin/gh/houseroad/9/base 2022-03-17T19:07:03.3218320Z * [new branch] gh/houseroad/9/orig -> origin/gh/houseroad/9/orig 2022-03-17T19:07:03.3220591Z * [new branch] gh/huiguoo/16/base -> origin/gh/huiguoo/16/base 2022-03-17T19:07:03.3223715Z * [new branch] gh/huiguoo/16/head -> origin/gh/huiguoo/16/head 2022-03-17T19:07:03.3225181Z * [new branch] gh/huiguoo/16/orig -> origin/gh/huiguoo/16/orig 2022-03-17T19:07:03.3227286Z * [new branch] gh/huiguoo/3/base -> origin/gh/huiguoo/3/base 2022-03-17T19:07:03.3228253Z * [new branch] gh/huiguoo/3/orig -> origin/gh/huiguoo/3/orig 2022-03-17T19:07:03.3230349Z * [new branch] gh/huiguoo/39/base -> origin/gh/huiguoo/39/base 2022-03-17T19:07:03.3231598Z * [new branch] gh/huiguoo/39/head -> origin/gh/huiguoo/39/head 2022-03-17T19:07:03.3232947Z * [new branch] gh/huiguoo/39/orig -> origin/gh/huiguoo/39/orig 2022-03-17T19:07:03.3235197Z * [new branch] gh/huiguoo/4/base -> origin/gh/huiguoo/4/base 2022-03-17T19:07:03.3236490Z * [new branch] gh/huiguoo/4/orig -> origin/gh/huiguoo/4/orig 2022-03-17T19:07:03.3238557Z * [new branch] gh/huiguoo/40/base -> origin/gh/huiguoo/40/base 2022-03-17T19:07:03.3239492Z * [new branch] gh/huiguoo/40/head -> origin/gh/huiguoo/40/head 2022-03-17T19:07:03.3240919Z * [new branch] gh/huiguoo/40/orig -> origin/gh/huiguoo/40/orig 2022-03-17T19:07:03.3243195Z * [new branch] gh/huiguoo/41/base -> origin/gh/huiguoo/41/base 2022-03-17T19:07:03.3244021Z * [new branch] gh/huiguoo/41/head -> origin/gh/huiguoo/41/head 2022-03-17T19:07:03.3245651Z * [new branch] gh/huiguoo/41/orig -> origin/gh/huiguoo/41/orig 2022-03-17T19:07:03.3247317Z * [new branch] gh/huiguoo/42/base -> origin/gh/huiguoo/42/base 2022-03-17T19:07:03.3248427Z * [new branch] gh/huiguoo/42/head -> origin/gh/huiguoo/42/head 2022-03-17T19:07:03.3249891Z * [new branch] gh/huiguoo/42/orig -> origin/gh/huiguoo/42/orig 2022-03-17T19:07:03.3251993Z * [new branch] gh/huiguoo/50/base -> origin/gh/huiguoo/50/base 2022-03-17T19:07:03.3253696Z * [new branch] gh/huiguoo/50/head -> origin/gh/huiguoo/50/head 2022-03-17T19:07:03.3254943Z * [new branch] gh/huiguoo/50/orig -> origin/gh/huiguoo/50/orig 2022-03-17T19:07:03.3256930Z * [new branch] gh/huiguoo/51/base -> origin/gh/huiguoo/51/base 2022-03-17T19:07:03.3258169Z * [new branch] gh/huiguoo/51/head -> origin/gh/huiguoo/51/head 2022-03-17T19:07:03.3259654Z * [new branch] gh/huiguoo/51/orig -> origin/gh/huiguoo/51/orig 2022-03-17T19:07:03.3261855Z * [new branch] gh/huiguoo/54/base -> origin/gh/huiguoo/54/base 2022-03-17T19:07:03.3262896Z * [new branch] gh/huiguoo/54/head -> origin/gh/huiguoo/54/head 2022-03-17T19:07:03.3264546Z * [new branch] gh/huiguoo/54/orig -> origin/gh/huiguoo/54/orig 2022-03-17T19:07:03.3266441Z * [new branch] gh/huiguoo/57/base -> origin/gh/huiguoo/57/base 2022-03-17T19:07:03.3267528Z * [new branch] gh/huiguoo/57/head -> origin/gh/huiguoo/57/head 2022-03-17T19:07:03.3269904Z * [new branch] gh/huiguoo/57/orig -> origin/gh/huiguoo/57/orig 2022-03-17T19:07:03.3272275Z * [new branch] gh/huiguoo/58/base -> origin/gh/huiguoo/58/base 2022-03-17T19:07:03.3273600Z * [new branch] gh/huiguoo/58/head -> origin/gh/huiguoo/58/head 2022-03-17T19:07:03.3275477Z * [new branch] gh/huiguoo/58/orig -> origin/gh/huiguoo/58/orig 2022-03-17T19:07:03.3277342Z * [new branch] gh/husthyc/10/base -> origin/gh/husthyc/10/base 2022-03-17T19:07:03.3278724Z * [new branch] gh/husthyc/10/head -> origin/gh/husthyc/10/head 2022-03-17T19:07:03.3280338Z * [new branch] gh/husthyc/10/orig -> origin/gh/husthyc/10/orig 2022-03-17T19:07:03.3282176Z * [new branch] gh/husthyc/2/base -> origin/gh/husthyc/2/base 2022-03-17T19:07:03.3283333Z * [new branch] gh/husthyc/2/orig -> origin/gh/husthyc/2/orig 2022-03-17T19:07:03.3285339Z * [new branch] gh/husthyc/22/base -> origin/gh/husthyc/22/base 2022-03-17T19:07:03.3286554Z * [new branch] gh/husthyc/22/head -> origin/gh/husthyc/22/head 2022-03-17T19:07:03.3288461Z * [new branch] gh/husthyc/25/base -> origin/gh/husthyc/25/base 2022-03-17T19:07:03.3289708Z * [new branch] gh/husthyc/25/orig -> origin/gh/husthyc/25/orig 2022-03-17T19:07:03.3291680Z * [new branch] gh/husthyc/26/base -> origin/gh/husthyc/26/base 2022-03-17T19:07:03.3293063Z * [new branch] gh/husthyc/26/orig -> origin/gh/husthyc/26/orig 2022-03-17T19:07:03.3295075Z * [new branch] gh/husthyc/27/base -> origin/gh/husthyc/27/base 2022-03-17T19:07:03.3296312Z * [new branch] gh/husthyc/27/orig -> origin/gh/husthyc/27/orig 2022-03-17T19:07:03.3298239Z * [new branch] gh/husthyc/3/orig -> origin/gh/husthyc/3/orig 2022-03-17T19:07:03.3300051Z * [new branch] gh/husthyc/32/base -> origin/gh/husthyc/32/base 2022-03-17T19:07:03.3301134Z * [new branch] gh/husthyc/32/head -> origin/gh/husthyc/32/head 2022-03-17T19:07:03.3303120Z * [new branch] gh/husthyc/32/orig -> origin/gh/husthyc/32/orig 2022-03-17T19:07:03.3305116Z * [new branch] gh/husthyc/33/base -> origin/gh/husthyc/33/base 2022-03-17T19:07:03.3306396Z * [new branch] gh/husthyc/33/head -> origin/gh/husthyc/33/head 2022-03-17T19:07:03.3307770Z * [new branch] gh/husthyc/33/orig -> origin/gh/husthyc/33/orig 2022-03-17T19:07:03.3309362Z * [new branch] gh/husthyc/34/base -> origin/gh/husthyc/34/base 2022-03-17T19:07:03.3310481Z * [new branch] gh/husthyc/34/head -> origin/gh/husthyc/34/head 2022-03-17T19:07:03.3312559Z * [new branch] gh/husthyc/34/orig -> origin/gh/husthyc/34/orig 2022-03-17T19:07:03.3314009Z * [new branch] gh/husthyc/4/base -> origin/gh/husthyc/4/base 2022-03-17T19:07:03.3316041Z * [new branch] gh/husthyc/5/base -> origin/gh/husthyc/5/base 2022-03-17T19:07:03.3317212Z * [new branch] gh/husthyc/6/base -> origin/gh/husthyc/6/base 2022-03-17T19:07:03.3318886Z * [new branch] gh/husthyc/7/base -> origin/gh/husthyc/7/base 2022-03-17T19:07:03.3320250Z * [new branch] gh/husthyc/8/base -> origin/gh/husthyc/8/base 2022-03-17T19:07:03.3322327Z * [new branch] gh/husthyc/9/base -> origin/gh/husthyc/9/base 2022-03-17T19:07:03.3324406Z * [new branch] gh/hx89/10/base -> origin/gh/hx89/10/base 2022-03-17T19:07:03.3325591Z * [new branch] gh/hx89/10/head -> origin/gh/hx89/10/head 2022-03-17T19:07:03.3327514Z * [new branch] gh/hx89/10/orig -> origin/gh/hx89/10/orig 2022-03-17T19:07:03.3328951Z * [new branch] gh/hx89/11/base -> origin/gh/hx89/11/base 2022-03-17T19:07:03.3330299Z * [new branch] gh/hx89/11/head -> origin/gh/hx89/11/head 2022-03-17T19:07:03.3331762Z * [new branch] gh/hx89/11/orig -> origin/gh/hx89/11/orig 2022-03-17T19:07:03.3333818Z * [new branch] gh/hx89/13/base -> origin/gh/hx89/13/base 2022-03-17T19:07:03.3335118Z * [new branch] gh/hx89/13/head -> origin/gh/hx89/13/head 2022-03-17T19:07:03.3336333Z * [new branch] gh/hx89/13/orig -> origin/gh/hx89/13/orig 2022-03-17T19:07:03.3338421Z * [new branch] gh/hx89/8/base -> origin/gh/hx89/8/base 2022-03-17T19:07:03.3339669Z * [new branch] gh/hx89/8/head -> origin/gh/hx89/8/head 2022-03-17T19:07:03.3341210Z * [new branch] gh/hx89/8/orig -> origin/gh/hx89/8/orig 2022-03-17T19:07:03.3343352Z * [new branch] gh/ifedan/114/base -> origin/gh/ifedan/114/base 2022-03-17T19:07:03.3344578Z * [new branch] gh/ifedan/114/head -> origin/gh/ifedan/114/head 2022-03-17T19:07:03.3346776Z * [new branch] gh/ifedan/115/base -> origin/gh/ifedan/115/base 2022-03-17T19:07:03.3348109Z * [new branch] gh/ifedan/115/head -> origin/gh/ifedan/115/head 2022-03-17T19:07:03.3349534Z * [new branch] gh/ifedan/116/base -> origin/gh/ifedan/116/base 2022-03-17T19:07:03.3350864Z * [new branch] gh/ifedan/116/head -> origin/gh/ifedan/116/head 2022-03-17T19:07:03.3352923Z * [new branch] gh/ifedan/117/base -> origin/gh/ifedan/117/base 2022-03-17T19:07:03.3353729Z * [new branch] gh/ifedan/117/head -> origin/gh/ifedan/117/head 2022-03-17T19:07:03.3355564Z * [new branch] gh/ifedan/68/base -> origin/gh/ifedan/68/base 2022-03-17T19:07:03.3356752Z * [new branch] gh/ifedan/68/head -> origin/gh/ifedan/68/head 2022-03-17T19:07:03.3358218Z * [new branch] gh/ifedan/68/orig -> origin/gh/ifedan/68/orig 2022-03-17T19:07:03.3360351Z * [new branch] gh/ifedan/69/base -> origin/gh/ifedan/69/base 2022-03-17T19:07:03.3361610Z * [new branch] gh/ifedan/69/head -> origin/gh/ifedan/69/head 2022-03-17T19:07:03.3363011Z * [new branch] gh/ifedan/69/orig -> origin/gh/ifedan/69/orig 2022-03-17T19:07:03.3364867Z * [new branch] gh/ifedan/70/base -> origin/gh/ifedan/70/base 2022-03-17T19:07:03.3366163Z * [new branch] gh/ifedan/70/head -> origin/gh/ifedan/70/head 2022-03-17T19:07:03.3367640Z * [new branch] gh/ifedan/70/orig -> origin/gh/ifedan/70/orig 2022-03-17T19:07:03.3369238Z * [new branch] gh/ifedan/71/base -> origin/gh/ifedan/71/base 2022-03-17T19:07:03.3370418Z * [new branch] gh/ifedan/71/head -> origin/gh/ifedan/71/head 2022-03-17T19:07:03.3372464Z * [new branch] gh/ifedan/71/orig -> origin/gh/ifedan/71/orig 2022-03-17T19:07:03.3373277Z * [new branch] gh/ifedan/72/base -> origin/gh/ifedan/72/base 2022-03-17T19:07:03.3374905Z * [new branch] gh/ifedan/72/head -> origin/gh/ifedan/72/head 2022-03-17T19:07:03.3376210Z * [new branch] gh/ifedan/72/orig -> origin/gh/ifedan/72/orig 2022-03-17T19:07:03.3377954Z * [new branch] gh/ifedan/73/base -> origin/gh/ifedan/73/base 2022-03-17T19:07:03.3379131Z * [new branch] gh/ifedan/73/head -> origin/gh/ifedan/73/head 2022-03-17T19:07:03.3380794Z * [new branch] gh/ifedan/73/orig -> origin/gh/ifedan/73/orig 2022-03-17T19:07:03.3382621Z * [new branch] gh/ifedan/74/base -> origin/gh/ifedan/74/base 2022-03-17T19:07:03.3383988Z * [new branch] gh/ifedan/74/head -> origin/gh/ifedan/74/head 2022-03-17T19:07:03.3385736Z * [new branch] gh/ifedan/74/orig -> origin/gh/ifedan/74/orig 2022-03-17T19:07:03.3387931Z * [new branch] gh/ilia-cher/39/base -> origin/gh/ilia-cher/39/base 2022-03-17T19:07:03.3388872Z * [new branch] gh/ilia-cher/39/orig -> origin/gh/ilia-cher/39/orig 2022-03-17T19:07:03.3391458Z * [new branch] gh/ilia-cher/50/base -> origin/gh/ilia-cher/50/base 2022-03-17T19:07:03.3392763Z * [new branch] gh/ilia-cher/50/head -> origin/gh/ilia-cher/50/head 2022-03-17T19:07:03.3394206Z * [new branch] gh/ilia-cher/51/base -> origin/gh/ilia-cher/51/base 2022-03-17T19:07:03.3395673Z * [new branch] gh/ilia-cher/51/head -> origin/gh/ilia-cher/51/head 2022-03-17T19:07:03.3397782Z * [new branch] gh/ilia-cher/53/base -> origin/gh/ilia-cher/53/base 2022-03-17T19:07:03.3399148Z * [new branch] gh/ilia-cher/53/head -> origin/gh/ilia-cher/53/head 2022-03-17T19:07:03.3400576Z * [new branch] gh/ilia-cher/53/orig -> origin/gh/ilia-cher/53/orig 2022-03-17T19:07:03.3403310Z * [new branch] gh/ilia-cher/83/base -> origin/gh/ilia-cher/83/base 2022-03-17T19:07:03.3404764Z * [new branch] gh/ilia-cher/83/head -> origin/gh/ilia-cher/83/head 2022-03-17T19:07:03.3406920Z * [new branch] gh/ilia-cher/84/base -> origin/gh/ilia-cher/84/base 2022-03-17T19:07:03.3408013Z * [new branch] gh/ilia-cher/84/head -> origin/gh/ilia-cher/84/head 2022-03-17T19:07:03.3409429Z * [new branch] gh/ilia-cher/84/orig -> origin/gh/ilia-cher/84/orig 2022-03-17T19:07:03.3411587Z * [new branch] gh/ilia-cher/86/base -> origin/gh/ilia-cher/86/base 2022-03-17T19:07:03.3413578Z * [new branch] gh/ilia-cher/86/head -> origin/gh/ilia-cher/86/head 2022-03-17T19:07:03.3414763Z * [new branch] gh/ilia-cher/86/orig -> origin/gh/ilia-cher/86/orig 2022-03-17T19:07:03.3416953Z * [new branch] gh/iramazanli/10/base -> origin/gh/iramazanli/10/base 2022-03-17T19:07:03.3418255Z * [new branch] gh/iramazanli/10/head -> origin/gh/iramazanli/10/head 2022-03-17T19:07:03.3419684Z * [new branch] gh/iramazanli/10/orig -> origin/gh/iramazanli/10/orig 2022-03-17T19:07:03.3422179Z * [new branch] gh/iramazanli/11/base -> origin/gh/iramazanli/11/base 2022-03-17T19:07:03.3423495Z * [new branch] gh/iramazanli/11/head -> origin/gh/iramazanli/11/head 2022-03-17T19:07:03.3424862Z * [new branch] gh/iramazanli/11/orig -> origin/gh/iramazanli/11/orig 2022-03-17T19:07:03.3426928Z * [new branch] gh/iramazanli/12/base -> origin/gh/iramazanli/12/base 2022-03-17T19:07:03.3428239Z * [new branch] gh/iramazanli/12/head -> origin/gh/iramazanli/12/head 2022-03-17T19:07:03.3429542Z * [new branch] gh/iramazanli/12/orig -> origin/gh/iramazanli/12/orig 2022-03-17T19:07:03.3432099Z * [new branch] gh/iramazanli/8/base -> origin/gh/iramazanli/8/base 2022-03-17T19:07:03.3433401Z * [new branch] gh/iramazanli/8/head -> origin/gh/iramazanli/8/head 2022-03-17T19:07:03.3434794Z * [new branch] gh/iramazanli/8/orig -> origin/gh/iramazanli/8/orig 2022-03-17T19:07:03.3436853Z * [new branch] gh/iramazanli/9/base -> origin/gh/iramazanli/9/base 2022-03-17T19:07:03.3437771Z * [new branch] gh/iramazanli/9/head -> origin/gh/iramazanli/9/head 2022-03-17T19:07:03.3439253Z * [new branch] gh/iramazanli/9/orig -> origin/gh/iramazanli/9/orig 2022-03-17T19:07:03.3441594Z * [new branch] gh/iseeyuan/1/base -> origin/gh/iseeyuan/1/base 2022-03-17T19:07:03.3443453Z * [new branch] gh/iseeyuan/10/base -> origin/gh/iseeyuan/10/base 2022-03-17T19:07:03.3444642Z * [new branch] gh/iseeyuan/10/head -> origin/gh/iseeyuan/10/head 2022-03-17T19:07:03.3446150Z * [new branch] gh/iseeyuan/10/orig -> origin/gh/iseeyuan/10/orig 2022-03-17T19:07:03.3448344Z * [new branch] gh/iseeyuan/101/base -> origin/gh/iseeyuan/101/base 2022-03-17T19:07:03.3449539Z * [new branch] gh/iseeyuan/101/head -> origin/gh/iseeyuan/101/head 2022-03-17T19:07:03.3451225Z * [new branch] gh/iseeyuan/101/orig -> origin/gh/iseeyuan/101/orig 2022-03-17T19:07:03.3452866Z * [new branch] gh/iseeyuan/110/base -> origin/gh/iseeyuan/110/base 2022-03-17T19:07:03.3454191Z * [new branch] gh/iseeyuan/110/head -> origin/gh/iseeyuan/110/head 2022-03-17T19:07:03.3455615Z * [new branch] gh/iseeyuan/110/orig -> origin/gh/iseeyuan/110/orig 2022-03-17T19:07:03.3457584Z * [new branch] gh/iseeyuan/112/base -> origin/gh/iseeyuan/112/base 2022-03-17T19:07:03.3459484Z * [new branch] gh/iseeyuan/112/head -> origin/gh/iseeyuan/112/head 2022-03-17T19:07:03.3460553Z * [new branch] gh/iseeyuan/112/orig -> origin/gh/iseeyuan/112/orig 2022-03-17T19:07:03.3463114Z * [new branch] gh/iseeyuan/114/base -> origin/gh/iseeyuan/114/base 2022-03-17T19:07:03.3464365Z * [new branch] gh/iseeyuan/114/head -> origin/gh/iseeyuan/114/head 2022-03-17T19:07:03.3466051Z * [new branch] gh/iseeyuan/114/orig -> origin/gh/iseeyuan/114/orig 2022-03-17T19:07:03.3467581Z * [new branch] gh/iseeyuan/116/base -> origin/gh/iseeyuan/116/base 2022-03-17T19:07:03.3469795Z * [new branch] gh/iseeyuan/116/head -> origin/gh/iseeyuan/116/head 2022-03-17T19:07:03.3471135Z * [new branch] gh/iseeyuan/116/orig -> origin/gh/iseeyuan/116/orig 2022-03-17T19:07:03.3473291Z * [new branch] gh/iseeyuan/118/base -> origin/gh/iseeyuan/118/base 2022-03-17T19:07:03.3474152Z * [new branch] gh/iseeyuan/118/head -> origin/gh/iseeyuan/118/head 2022-03-17T19:07:03.3475761Z * [new branch] gh/iseeyuan/118/orig -> origin/gh/iseeyuan/118/orig 2022-03-17T19:07:03.3477690Z * [new branch] gh/iseeyuan/119/base -> origin/gh/iseeyuan/119/base 2022-03-17T19:07:03.3478596Z * [new branch] gh/iseeyuan/119/head -> origin/gh/iseeyuan/119/head 2022-03-17T19:07:03.3480063Z * [new branch] gh/iseeyuan/119/orig -> origin/gh/iseeyuan/119/orig 2022-03-17T19:07:03.3482644Z * [new branch] gh/iseeyuan/120/base -> origin/gh/iseeyuan/120/base 2022-03-17T19:07:03.3484070Z * [new branch] gh/iseeyuan/120/head -> origin/gh/iseeyuan/120/head 2022-03-17T19:07:03.3485556Z * [new branch] gh/iseeyuan/120/orig -> origin/gh/iseeyuan/120/orig 2022-03-17T19:07:03.3487712Z * [new branch] gh/iseeyuan/122/base -> origin/gh/iseeyuan/122/base 2022-03-17T19:07:03.3489296Z * [new branch] gh/iseeyuan/122/head -> origin/gh/iseeyuan/122/head 2022-03-17T19:07:03.3490564Z * [new branch] gh/iseeyuan/122/orig -> origin/gh/iseeyuan/122/orig 2022-03-17T19:07:03.3492717Z * [new branch] gh/iseeyuan/125/base -> origin/gh/iseeyuan/125/base 2022-03-17T19:07:03.3493662Z * [new branch] gh/iseeyuan/125/head -> origin/gh/iseeyuan/125/head 2022-03-17T19:07:03.3495530Z * [new branch] gh/iseeyuan/125/orig -> origin/gh/iseeyuan/125/orig 2022-03-17T19:07:03.3497511Z * [new branch] gh/iseeyuan/128/base -> origin/gh/iseeyuan/128/base 2022-03-17T19:07:03.3498248Z * [new branch] gh/iseeyuan/128/head -> origin/gh/iseeyuan/128/head 2022-03-17T19:07:03.3500197Z * [new branch] gh/iseeyuan/128/orig -> origin/gh/iseeyuan/128/orig 2022-03-17T19:07:03.3501718Z * [new branch] gh/iseeyuan/130/base -> origin/gh/iseeyuan/130/base 2022-03-17T19:07:03.3503816Z * [new branch] gh/iseeyuan/130/head -> origin/gh/iseeyuan/130/head 2022-03-17T19:07:03.3504795Z * [new branch] gh/iseeyuan/130/orig -> origin/gh/iseeyuan/130/orig 2022-03-17T19:07:03.3506873Z * [new branch] gh/iseeyuan/133/base -> origin/gh/iseeyuan/133/base 2022-03-17T19:07:03.3507999Z * [new branch] gh/iseeyuan/133/head -> origin/gh/iseeyuan/133/head 2022-03-17T19:07:03.3509415Z * [new branch] gh/iseeyuan/133/orig -> origin/gh/iseeyuan/133/orig 2022-03-17T19:07:03.3511436Z * [new branch] gh/iseeyuan/134/base -> origin/gh/iseeyuan/134/base 2022-03-17T19:07:03.3512591Z * [new branch] gh/iseeyuan/134/head -> origin/gh/iseeyuan/134/head 2022-03-17T19:07:03.3513946Z * [new branch] gh/iseeyuan/134/orig -> origin/gh/iseeyuan/134/orig 2022-03-17T19:07:03.3515855Z * [new branch] gh/iseeyuan/18/base -> origin/gh/iseeyuan/18/base 2022-03-17T19:07:03.3517036Z * [new branch] gh/iseeyuan/18/head -> origin/gh/iseeyuan/18/head 2022-03-17T19:07:03.3518401Z * [new branch] gh/iseeyuan/18/orig -> origin/gh/iseeyuan/18/orig 2022-03-17T19:07:03.3520446Z * [new branch] gh/iseeyuan/2/base -> origin/gh/iseeyuan/2/base 2022-03-17T19:07:03.3521790Z * [new branch] gh/iseeyuan/2/head -> origin/gh/iseeyuan/2/head 2022-03-17T19:07:03.3523818Z * [new branch] gh/iseeyuan/2/orig -> origin/gh/iseeyuan/2/orig 2022-03-17T19:07:03.3525287Z * [new branch] gh/iseeyuan/20/base -> origin/gh/iseeyuan/20/base 2022-03-17T19:07:03.3526602Z * [new branch] gh/iseeyuan/20/head -> origin/gh/iseeyuan/20/head 2022-03-17T19:07:03.3527959Z * [new branch] gh/iseeyuan/20/orig -> origin/gh/iseeyuan/20/orig 2022-03-17T19:07:03.3530108Z * [new branch] gh/iseeyuan/22/base -> origin/gh/iseeyuan/22/base 2022-03-17T19:07:03.3531161Z * [new branch] gh/iseeyuan/22/head -> origin/gh/iseeyuan/22/head 2022-03-17T19:07:03.3532586Z * [new branch] gh/iseeyuan/22/orig -> origin/gh/iseeyuan/22/orig 2022-03-17T19:07:03.3534577Z * [new branch] gh/iseeyuan/25/base -> origin/gh/iseeyuan/25/base 2022-03-17T19:07:03.3535709Z * [new branch] gh/iseeyuan/25/head -> origin/gh/iseeyuan/25/head 2022-03-17T19:07:03.3537053Z * [new branch] gh/iseeyuan/25/orig -> origin/gh/iseeyuan/25/orig 2022-03-17T19:07:03.3539086Z * [new branch] gh/iseeyuan/29/base -> origin/gh/iseeyuan/29/base 2022-03-17T19:07:03.3540205Z * [new branch] gh/iseeyuan/29/head -> origin/gh/iseeyuan/29/head 2022-03-17T19:07:03.3542097Z * [new branch] gh/iseeyuan/29/orig -> origin/gh/iseeyuan/29/orig 2022-03-17T19:07:03.3544119Z * [new branch] gh/iseeyuan/3/base -> origin/gh/iseeyuan/3/base 2022-03-17T19:07:03.3545281Z * [new branch] gh/iseeyuan/3/head -> origin/gh/iseeyuan/3/head 2022-03-17T19:07:03.3546603Z * [new branch] gh/iseeyuan/3/orig -> origin/gh/iseeyuan/3/orig 2022-03-17T19:07:03.3548627Z * [new branch] gh/iseeyuan/30/base -> origin/gh/iseeyuan/30/base 2022-03-17T19:07:03.3549695Z * [new branch] gh/iseeyuan/30/head -> origin/gh/iseeyuan/30/head 2022-03-17T19:07:03.3551331Z * [new branch] gh/iseeyuan/30/orig -> origin/gh/iseeyuan/30/orig 2022-03-17T19:07:03.3553510Z * [new branch] gh/iseeyuan/38/base -> origin/gh/iseeyuan/38/base 2022-03-17T19:07:03.3554659Z * [new branch] gh/iseeyuan/38/head -> origin/gh/iseeyuan/38/head 2022-03-17T19:07:03.3556258Z * [new branch] gh/iseeyuan/38/orig -> origin/gh/iseeyuan/38/orig 2022-03-17T19:07:03.3557664Z * [new branch] gh/iseeyuan/4/base -> origin/gh/iseeyuan/4/base 2022-03-17T19:07:03.3559060Z * [new branch] gh/iseeyuan/4/head -> origin/gh/iseeyuan/4/head 2022-03-17T19:07:03.3560346Z * [new branch] gh/iseeyuan/4/orig -> origin/gh/iseeyuan/4/orig 2022-03-17T19:07:03.3562506Z * [new branch] gh/iseeyuan/48/base -> origin/gh/iseeyuan/48/base 2022-03-17T19:07:03.3563688Z * [new branch] gh/iseeyuan/48/head -> origin/gh/iseeyuan/48/head 2022-03-17T19:07:03.3565336Z * [new branch] gh/iseeyuan/48/orig -> origin/gh/iseeyuan/48/orig 2022-03-17T19:07:03.3566738Z * [new branch] gh/iseeyuan/5/base -> origin/gh/iseeyuan/5/base 2022-03-17T19:07:03.3567979Z * [new branch] gh/iseeyuan/5/head -> origin/gh/iseeyuan/5/head 2022-03-17T19:07:03.3569779Z * [new branch] gh/iseeyuan/5/orig -> origin/gh/iseeyuan/5/orig 2022-03-17T19:07:03.3571522Z * [new branch] gh/iseeyuan/52/base -> origin/gh/iseeyuan/52/base 2022-03-17T19:07:03.3572727Z * [new branch] gh/iseeyuan/52/head -> origin/gh/iseeyuan/52/head 2022-03-17T19:07:03.3574419Z * [new branch] gh/iseeyuan/52/orig -> origin/gh/iseeyuan/52/orig 2022-03-17T19:07:03.3576582Z * [new branch] gh/iseeyuan/56/base -> origin/gh/iseeyuan/56/base 2022-03-17T19:07:03.3577728Z * [new branch] gh/iseeyuan/56/head -> origin/gh/iseeyuan/56/head 2022-03-17T19:07:03.3579320Z * [new branch] gh/iseeyuan/56/orig -> origin/gh/iseeyuan/56/orig 2022-03-17T19:07:03.3580965Z * [new branch] gh/iseeyuan/6/base -> origin/gh/iseeyuan/6/base 2022-03-17T19:07:03.3582851Z * [new branch] gh/iseeyuan/6/head -> origin/gh/iseeyuan/6/head 2022-03-17T19:07:03.3584212Z * [new branch] gh/iseeyuan/6/orig -> origin/gh/iseeyuan/6/orig 2022-03-17T19:07:03.3586172Z * [new branch] gh/iseeyuan/62/base -> origin/gh/iseeyuan/62/base 2022-03-17T19:07:03.3587319Z * [new branch] gh/iseeyuan/62/head -> origin/gh/iseeyuan/62/head 2022-03-17T19:07:03.3589008Z * [new branch] gh/iseeyuan/62/orig -> origin/gh/iseeyuan/62/orig 2022-03-17T19:07:03.3591485Z * [new branch] gh/iseeyuan/65/base -> origin/gh/iseeyuan/65/base 2022-03-17T19:07:03.3593695Z * [new branch] gh/iseeyuan/65/head -> origin/gh/iseeyuan/65/head 2022-03-17T19:07:03.3595070Z * [new branch] gh/iseeyuan/65/orig -> origin/gh/iseeyuan/65/orig 2022-03-17T19:07:03.3596795Z * [new branch] gh/iseeyuan/66/base -> origin/gh/iseeyuan/66/base 2022-03-17T19:07:03.3597993Z * [new branch] gh/iseeyuan/66/head -> origin/gh/iseeyuan/66/head 2022-03-17T19:07:03.3599688Z * [new branch] gh/iseeyuan/66/orig -> origin/gh/iseeyuan/66/orig 2022-03-17T19:07:03.3601528Z * [new branch] gh/iseeyuan/73/base -> origin/gh/iseeyuan/73/base 2022-03-17T19:07:03.3602789Z * [new branch] gh/iseeyuan/73/head -> origin/gh/iseeyuan/73/head 2022-03-17T19:07:03.3604512Z * [new branch] gh/iseeyuan/73/orig -> origin/gh/iseeyuan/73/orig 2022-03-17T19:07:03.3606299Z * [new branch] gh/iseeyuan/77/base -> origin/gh/iseeyuan/77/base 2022-03-17T19:07:03.3607447Z * [new branch] gh/iseeyuan/77/head -> origin/gh/iseeyuan/77/head 2022-03-17T19:07:03.3609076Z * [new branch] gh/iseeyuan/77/orig -> origin/gh/iseeyuan/77/orig 2022-03-17T19:07:03.3610775Z * [new branch] gh/iseeyuan/81/base -> origin/gh/iseeyuan/81/base 2022-03-17T19:07:03.3612034Z * [new branch] gh/iseeyuan/81/head -> origin/gh/iseeyuan/81/head 2022-03-17T19:07:03.3613649Z * [new branch] gh/iseeyuan/81/orig -> origin/gh/iseeyuan/81/orig 2022-03-17T19:07:03.3615254Z * [new branch] gh/iseeyuan/82/base -> origin/gh/iseeyuan/82/base 2022-03-17T19:07:03.3616464Z * [new branch] gh/iseeyuan/82/head -> origin/gh/iseeyuan/82/head 2022-03-17T19:07:03.3617966Z * [new branch] gh/iseeyuan/82/orig -> origin/gh/iseeyuan/82/orig 2022-03-17T19:07:03.3619890Z * [new branch] gh/iseeyuan/83/base -> origin/gh/iseeyuan/83/base 2022-03-17T19:07:03.3622097Z * [new branch] gh/iseeyuan/83/head -> origin/gh/iseeyuan/83/head 2022-03-17T19:07:03.3623247Z * [new branch] gh/iseeyuan/83/orig -> origin/gh/iseeyuan/83/orig 2022-03-17T19:07:03.3625496Z * [new branch] gh/iseeyuan/90/base -> origin/gh/iseeyuan/90/base 2022-03-17T19:07:03.3626659Z * [new branch] gh/iseeyuan/90/head -> origin/gh/iseeyuan/90/head 2022-03-17T19:07:03.3628268Z * [new branch] gh/iseeyuan/90/orig -> origin/gh/iseeyuan/90/orig 2022-03-17T19:07:03.3630622Z * [new branch] gh/iseeyuan/93/base -> origin/gh/iseeyuan/93/base 2022-03-17T19:07:03.3632003Z * [new branch] gh/iseeyuan/93/head -> origin/gh/iseeyuan/93/head 2022-03-17T19:07:03.3633252Z * [new branch] gh/iseeyuan/93/orig -> origin/gh/iseeyuan/93/orig 2022-03-17T19:07:03.3635349Z * [new branch] gh/iseeyuan/95/base -> origin/gh/iseeyuan/95/base 2022-03-17T19:07:03.3636500Z * [new branch] gh/iseeyuan/95/head -> origin/gh/iseeyuan/95/head 2022-03-17T19:07:03.3638136Z * [new branch] gh/iseeyuan/95/orig -> origin/gh/iseeyuan/95/orig 2022-03-17T19:07:03.3639702Z * [new branch] gh/iseeyuan/97/base -> origin/gh/iseeyuan/97/base 2022-03-17T19:07:03.3640885Z * [new branch] gh/iseeyuan/97/head -> origin/gh/iseeyuan/97/head 2022-03-17T19:07:03.3642557Z * [new branch] gh/iseeyuan/97/orig -> origin/gh/iseeyuan/97/orig 2022-03-17T19:07:03.3644511Z * [new branch] gh/iseeyuan/98/base -> origin/gh/iseeyuan/98/base 2022-03-17T19:07:03.3645560Z * [new branch] gh/iseeyuan/98/head -> origin/gh/iseeyuan/98/head 2022-03-17T19:07:03.3647209Z * [new branch] gh/iseeyuan/98/orig -> origin/gh/iseeyuan/98/orig 2022-03-17T19:07:03.3649420Z * [new branch] gh/ivankobzarev/1/base -> origin/gh/ivankobzarev/1/base 2022-03-17T19:07:03.3650606Z * [new branch] gh/ivankobzarev/1/head -> origin/gh/ivankobzarev/1/head 2022-03-17T19:07:03.3652635Z * [new branch] gh/ivankobzarev/102/base -> origin/gh/ivankobzarev/102/base 2022-03-17T19:07:03.3653928Z * [new branch] gh/ivankobzarev/102/head -> origin/gh/ivankobzarev/102/head 2022-03-17T19:07:03.3655562Z * [new branch] gh/ivankobzarev/102/orig -> origin/gh/ivankobzarev/102/orig 2022-03-17T19:07:03.3657325Z * [new branch] gh/ivankobzarev/103/base -> origin/gh/ivankobzarev/103/base 2022-03-17T19:07:03.3658371Z * [new branch] gh/ivankobzarev/103/head -> origin/gh/ivankobzarev/103/head 2022-03-17T19:07:03.3660223Z * [new branch] gh/ivankobzarev/103/orig -> origin/gh/ivankobzarev/103/orig 2022-03-17T19:07:03.3661677Z * [new branch] gh/ivankobzarev/104/base -> origin/gh/ivankobzarev/104/base 2022-03-17T19:07:03.3663525Z * [new branch] gh/ivankobzarev/104/head -> origin/gh/ivankobzarev/104/head 2022-03-17T19:07:03.3664693Z * [new branch] gh/ivankobzarev/104/orig -> origin/gh/ivankobzarev/104/orig 2022-03-17T19:07:03.3666685Z * [new branch] gh/ivankobzarev/106/base -> origin/gh/ivankobzarev/106/base 2022-03-17T19:07:03.3667861Z * [new branch] gh/ivankobzarev/106/head -> origin/gh/ivankobzarev/106/head 2022-03-17T19:07:03.3669495Z * [new branch] gh/ivankobzarev/106/orig -> origin/gh/ivankobzarev/106/orig 2022-03-17T19:07:03.3671198Z * [new branch] gh/ivankobzarev/107/base -> origin/gh/ivankobzarev/107/base 2022-03-17T19:07:03.3672585Z * [new branch] gh/ivankobzarev/107/head -> origin/gh/ivankobzarev/107/head 2022-03-17T19:07:03.3673835Z * [new branch] gh/ivankobzarev/107/orig -> origin/gh/ivankobzarev/107/orig 2022-03-17T19:07:03.3675848Z * [new branch] gh/ivankobzarev/114/base -> origin/gh/ivankobzarev/114/base 2022-03-17T19:07:03.3677158Z * [new branch] gh/ivankobzarev/114/head -> origin/gh/ivankobzarev/114/head 2022-03-17T19:07:03.3678534Z * [new branch] gh/ivankobzarev/114/orig -> origin/gh/ivankobzarev/114/orig 2022-03-17T19:07:03.3680389Z * [new branch] gh/ivankobzarev/115/base -> origin/gh/ivankobzarev/115/base 2022-03-17T19:07:03.3681580Z * [new branch] gh/ivankobzarev/115/head -> origin/gh/ivankobzarev/115/head 2022-03-17T19:07:03.3683381Z * [new branch] gh/ivankobzarev/115/orig -> origin/gh/ivankobzarev/115/orig 2022-03-17T19:07:03.3685198Z * [new branch] gh/ivankobzarev/12/base -> origin/gh/ivankobzarev/12/base 2022-03-17T19:07:03.3686368Z * [new branch] gh/ivankobzarev/12/head -> origin/gh/ivankobzarev/12/head 2022-03-17T19:07:03.3687760Z * [new branch] gh/ivankobzarev/12/orig -> origin/gh/ivankobzarev/12/orig 2022-03-17T19:07:03.3689448Z * [new branch] gh/ivankobzarev/2/base -> origin/gh/ivankobzarev/2/base 2022-03-17T19:07:03.3690762Z * [new branch] gh/ivankobzarev/2/head -> origin/gh/ivankobzarev/2/head 2022-03-17T19:07:03.3692532Z * [new branch] gh/ivankobzarev/3/base -> origin/gh/ivankobzarev/3/base 2022-03-17T19:07:03.3693552Z * [new branch] gh/ivankobzarev/3/head -> origin/gh/ivankobzarev/3/head 2022-03-17T19:07:03.3695628Z * [new branch] gh/ivankobzarev/39/base -> origin/gh/ivankobzarev/39/base 2022-03-17T19:07:03.3696791Z * [new branch] gh/ivankobzarev/39/head -> origin/gh/ivankobzarev/39/head 2022-03-17T19:07:03.3698487Z * [new branch] gh/ivankobzarev/39/orig -> origin/gh/ivankobzarev/39/orig 2022-03-17T19:07:03.3700081Z * [new branch] gh/ivankobzarev/4/base -> origin/gh/ivankobzarev/4/base 2022-03-17T19:07:03.3702463Z * [new branch] gh/ivankobzarev/4/head -> origin/gh/ivankobzarev/4/head 2022-03-17T19:07:03.3704320Z * [new branch] gh/ivankobzarev/41/base -> origin/gh/ivankobzarev/41/base 2022-03-17T19:07:03.3705465Z * [new branch] gh/ivankobzarev/41/head -> origin/gh/ivankobzarev/41/head 2022-03-17T19:07:03.3707089Z * [new branch] gh/ivankobzarev/41/orig -> origin/gh/ivankobzarev/41/orig 2022-03-17T19:07:03.3708902Z * [new branch] gh/ivankobzarev/44/base -> origin/gh/ivankobzarev/44/base 2022-03-17T19:07:03.3710033Z * [new branch] gh/ivankobzarev/44/head -> origin/gh/ivankobzarev/44/head 2022-03-17T19:07:03.3711777Z * [new branch] gh/ivankobzarev/44/orig -> origin/gh/ivankobzarev/44/orig 2022-03-17T19:07:03.3713018Z * [new branch] gh/ivankobzarev/45/base -> origin/gh/ivankobzarev/45/base 2022-03-17T19:07:03.3714323Z * [new branch] gh/ivankobzarev/45/head -> origin/gh/ivankobzarev/45/head 2022-03-17T19:07:03.3715505Z * [new branch] gh/ivankobzarev/45/orig -> origin/gh/ivankobzarev/45/orig 2022-03-17T19:07:03.3717446Z * [new branch] gh/ivankobzarev/49/base -> origin/gh/ivankobzarev/49/base 2022-03-17T19:07:03.3718604Z * [new branch] gh/ivankobzarev/49/head -> origin/gh/ivankobzarev/49/head 2022-03-17T19:07:03.3720229Z * [new branch] gh/ivankobzarev/49/orig -> origin/gh/ivankobzarev/49/orig 2022-03-17T19:07:03.3721913Z * [new branch] gh/ivankobzarev/5/base -> origin/gh/ivankobzarev/5/base 2022-03-17T19:07:03.3722967Z * [new branch] gh/ivankobzarev/5/head -> origin/gh/ivankobzarev/5/head 2022-03-17T19:07:03.3724928Z * [new branch] gh/ivankobzarev/50/base -> origin/gh/ivankobzarev/50/base 2022-03-17T19:07:03.3726159Z * [new branch] gh/ivankobzarev/50/head -> origin/gh/ivankobzarev/50/head 2022-03-17T19:07:03.3727484Z * [new branch] gh/ivankobzarev/50/orig -> origin/gh/ivankobzarev/50/orig 2022-03-17T19:07:03.3729402Z * [new branch] gh/ivankobzarev/51/base -> origin/gh/ivankobzarev/51/base 2022-03-17T19:07:03.3730592Z * [new branch] gh/ivankobzarev/51/head -> origin/gh/ivankobzarev/51/head 2022-03-17T19:07:03.3732170Z * [new branch] gh/ivankobzarev/51/orig -> origin/gh/ivankobzarev/51/orig 2022-03-17T19:07:03.3734069Z * [new branch] gh/ivankobzarev/52/base -> origin/gh/ivankobzarev/52/base 2022-03-17T19:07:03.3735251Z * [new branch] gh/ivankobzarev/52/head -> origin/gh/ivankobzarev/52/head 2022-03-17T19:07:03.3736988Z * [new branch] gh/ivankobzarev/52/orig -> origin/gh/ivankobzarev/52/orig 2022-03-17T19:07:03.3738393Z * [new branch] gh/ivankobzarev/53/base -> origin/gh/ivankobzarev/53/base 2022-03-17T19:07:03.3740156Z * [new branch] gh/ivankobzarev/53/head -> origin/gh/ivankobzarev/53/head 2022-03-17T19:07:03.3741319Z * [new branch] gh/ivankobzarev/53/orig -> origin/gh/ivankobzarev/53/orig 2022-03-17T19:07:03.3743161Z * [new branch] gh/ivankobzarev/54/base -> origin/gh/ivankobzarev/54/base 2022-03-17T19:07:03.3744292Z * [new branch] gh/ivankobzarev/54/head -> origin/gh/ivankobzarev/54/head 2022-03-17T19:07:03.3745899Z * [new branch] gh/ivankobzarev/54/orig -> origin/gh/ivankobzarev/54/orig 2022-03-17T19:07:03.3748119Z * [new branch] gh/ivankobzarev/55/base -> origin/gh/ivankobzarev/55/base 2022-03-17T19:07:03.3749288Z * [new branch] gh/ivankobzarev/55/head -> origin/gh/ivankobzarev/55/head 2022-03-17T19:07:03.3750912Z * [new branch] gh/ivankobzarev/55/orig -> origin/gh/ivankobzarev/55/orig 2022-03-17T19:07:03.3752309Z * [new branch] gh/ivankobzarev/56/base -> origin/gh/ivankobzarev/56/base 2022-03-17T19:07:03.3753586Z * [new branch] gh/ivankobzarev/56/head -> origin/gh/ivankobzarev/56/head 2022-03-17T19:07:03.3755689Z * [new branch] gh/ivankobzarev/57/base -> origin/gh/ivankobzarev/57/base 2022-03-17T19:07:03.3756714Z * [new branch] gh/ivankobzarev/57/head -> origin/gh/ivankobzarev/57/head 2022-03-17T19:07:03.3758370Z * [new branch] gh/ivankobzarev/57/orig -> origin/gh/ivankobzarev/57/orig 2022-03-17T19:07:03.3760943Z * [new branch] gh/ivankobzarev/58/base -> origin/gh/ivankobzarev/58/base 2022-03-17T19:07:03.3762114Z * [new branch] gh/ivankobzarev/58/head -> origin/gh/ivankobzarev/58/head 2022-03-17T19:07:03.3763716Z * [new branch] gh/ivankobzarev/58/orig -> origin/gh/ivankobzarev/58/orig 2022-03-17T19:07:03.3765380Z * [new branch] gh/ivankobzarev/59/base -> origin/gh/ivankobzarev/59/base 2022-03-17T19:07:03.3766621Z * [new branch] gh/ivankobzarev/59/head -> origin/gh/ivankobzarev/59/head 2022-03-17T19:07:03.3767852Z * [new branch] gh/ivankobzarev/59/orig -> origin/gh/ivankobzarev/59/orig 2022-03-17T19:07:03.3769671Z * [new branch] gh/ivankobzarev/6/base -> origin/gh/ivankobzarev/6/base 2022-03-17T19:07:03.3770819Z * [new branch] gh/ivankobzarev/6/head -> origin/gh/ivankobzarev/6/head 2022-03-17T19:07:03.3772777Z * [new branch] gh/ivankobzarev/62/base -> origin/gh/ivankobzarev/62/base 2022-03-17T19:07:03.3773805Z * [new branch] gh/ivankobzarev/62/head -> origin/gh/ivankobzarev/62/head 2022-03-17T19:07:03.3775480Z * [new branch] gh/ivankobzarev/62/orig -> origin/gh/ivankobzarev/62/orig 2022-03-17T19:07:03.3777260Z * [new branch] gh/ivankobzarev/63/base -> origin/gh/ivankobzarev/63/base 2022-03-17T19:07:03.3778785Z * [new branch] gh/ivankobzarev/63/head -> origin/gh/ivankobzarev/63/head 2022-03-17T19:07:03.3780185Z * [new branch] gh/ivankobzarev/63/orig -> origin/gh/ivankobzarev/63/orig 2022-03-17T19:07:03.3782213Z * [new branch] gh/ivankobzarev/64/base -> origin/gh/ivankobzarev/64/base 2022-03-17T19:07:03.3783332Z * [new branch] gh/ivankobzarev/64/head -> origin/gh/ivankobzarev/64/head 2022-03-17T19:07:03.3784995Z * [new branch] gh/ivankobzarev/64/orig -> origin/gh/ivankobzarev/64/orig 2022-03-17T19:07:03.3787399Z * [new branch] gh/ivankobzarev/66/base -> origin/gh/ivankobzarev/66/base 2022-03-17T19:07:03.3789175Z * [new branch] gh/ivankobzarev/66/head -> origin/gh/ivankobzarev/66/head 2022-03-17T19:07:03.3790480Z * [new branch] gh/ivankobzarev/66/orig -> origin/gh/ivankobzarev/66/orig 2022-03-17T19:07:03.3792657Z * [new branch] gh/ivankobzarev/67/base -> origin/gh/ivankobzarev/67/base 2022-03-17T19:07:03.3793820Z * [new branch] gh/ivankobzarev/67/head -> origin/gh/ivankobzarev/67/head 2022-03-17T19:07:03.3795613Z * [new branch] gh/ivankobzarev/67/orig -> origin/gh/ivankobzarev/67/orig 2022-03-17T19:07:03.3797263Z * [new branch] gh/ivankobzarev/68/base -> origin/gh/ivankobzarev/68/base 2022-03-17T19:07:03.3798566Z * [new branch] gh/ivankobzarev/68/head -> origin/gh/ivankobzarev/68/head 2022-03-17T19:07:03.3800931Z * [new branch] gh/ivankobzarev/68/orig -> origin/gh/ivankobzarev/68/orig 2022-03-17T19:07:03.3802784Z * [new branch] gh/ivankobzarev/69/base -> origin/gh/ivankobzarev/69/base 2022-03-17T19:07:03.3803922Z * [new branch] gh/ivankobzarev/69/head -> origin/gh/ivankobzarev/69/head 2022-03-17T19:07:03.3805242Z * [new branch] gh/ivankobzarev/69/orig -> origin/gh/ivankobzarev/69/orig 2022-03-17T19:07:03.3807075Z * [new branch] gh/ivankobzarev/7/base -> origin/gh/ivankobzarev/7/base 2022-03-17T19:07:03.3808087Z * [new branch] gh/ivankobzarev/7/head -> origin/gh/ivankobzarev/7/head 2022-03-17T19:07:03.3810136Z * [new branch] gh/ivankobzarev/70/base -> origin/gh/ivankobzarev/70/base 2022-03-17T19:07:03.3811210Z * [new branch] gh/ivankobzarev/70/head -> origin/gh/ivankobzarev/70/head 2022-03-17T19:07:03.3812917Z * [new branch] gh/ivankobzarev/70/orig -> origin/gh/ivankobzarev/70/orig 2022-03-17T19:07:03.3814389Z * [new branch] gh/ivankobzarev/71/base -> origin/gh/ivankobzarev/71/base 2022-03-17T19:07:03.3815692Z * [new branch] gh/ivankobzarev/71/head -> origin/gh/ivankobzarev/71/head 2022-03-17T19:07:03.3817219Z * [new branch] gh/ivankobzarev/71/orig -> origin/gh/ivankobzarev/71/orig 2022-03-17T19:07:03.3818936Z * [new branch] gh/ivankobzarev/72/base -> origin/gh/ivankobzarev/72/base 2022-03-17T19:07:03.3819973Z * [new branch] gh/ivankobzarev/72/head -> origin/gh/ivankobzarev/72/head 2022-03-17T19:07:03.3822144Z * [new branch] gh/ivankobzarev/72/orig -> origin/gh/ivankobzarev/72/orig 2022-03-17T19:07:03.3828293Z * [new branch] gh/ivankobzarev/73/base -> origin/gh/ivankobzarev/73/base 2022-03-17T19:07:03.3829533Z * [new branch] gh/ivankobzarev/73/head -> origin/gh/ivankobzarev/73/head 2022-03-17T19:07:03.3831013Z * [new branch] gh/ivankobzarev/73/orig -> origin/gh/ivankobzarev/73/orig 2022-03-17T19:07:03.3832728Z * [new branch] gh/ivankobzarev/74/base -> origin/gh/ivankobzarev/74/base 2022-03-17T19:07:03.3833814Z * [new branch] gh/ivankobzarev/74/head -> origin/gh/ivankobzarev/74/head 2022-03-17T19:07:03.3835136Z * [new branch] gh/ivankobzarev/74/orig -> origin/gh/ivankobzarev/74/orig 2022-03-17T19:07:03.3836951Z * [new branch] gh/ivankobzarev/8/base -> origin/gh/ivankobzarev/8/base 2022-03-17T19:07:03.3838064Z * [new branch] gh/ivankobzarev/8/head -> origin/gh/ivankobzarev/8/head 2022-03-17T19:07:03.3840054Z * [new branch] gh/ivankobzarev/83/base -> origin/gh/ivankobzarev/83/base 2022-03-17T19:07:03.3841220Z * [new branch] gh/ivankobzarev/83/head -> origin/gh/ivankobzarev/83/head 2022-03-17T19:07:03.3842801Z * [new branch] gh/ivankobzarev/83/orig -> origin/gh/ivankobzarev/83/orig 2022-03-17T19:07:03.3845033Z * [new branch] gh/ivankobzarev/84/base -> origin/gh/ivankobzarev/84/base 2022-03-17T19:07:03.3846144Z * [new branch] gh/ivankobzarev/84/head -> origin/gh/ivankobzarev/84/head 2022-03-17T19:07:03.3847540Z * [new branch] gh/ivankobzarev/84/orig -> origin/gh/ivankobzarev/84/orig 2022-03-17T19:07:03.3849618Z * [new branch] gh/ivankobzarev/85/base -> origin/gh/ivankobzarev/85/base 2022-03-17T19:07:03.3850799Z * [new branch] gh/ivankobzarev/85/head -> origin/gh/ivankobzarev/85/head 2022-03-17T19:07:03.3852165Z * [new branch] gh/ivankobzarev/85/orig -> origin/gh/ivankobzarev/85/orig 2022-03-17T19:07:03.3854331Z * [new branch] gh/ivankobzarev/86/base -> origin/gh/ivankobzarev/86/base 2022-03-17T19:07:03.3855443Z * [new branch] gh/ivankobzarev/86/head -> origin/gh/ivankobzarev/86/head 2022-03-17T19:07:03.3856779Z * [new branch] gh/ivankobzarev/86/orig -> origin/gh/ivankobzarev/86/orig 2022-03-17T19:07:03.3859447Z * [new branch] gh/ivankobzarev/87/base -> origin/gh/ivankobzarev/87/base 2022-03-17T19:07:03.3860801Z * [new branch] gh/ivankobzarev/87/head -> origin/gh/ivankobzarev/87/head 2022-03-17T19:07:03.3863082Z * [new branch] gh/ivankobzarev/87/orig -> origin/gh/ivankobzarev/87/orig 2022-03-17T19:07:03.3864729Z * [new branch] gh/ivankobzarev/90/base -> origin/gh/ivankobzarev/90/base 2022-03-17T19:07:03.3866141Z * [new branch] gh/ivankobzarev/90/head -> origin/gh/ivankobzarev/90/head 2022-03-17T19:07:03.3867437Z * [new branch] gh/ivankobzarev/90/orig -> origin/gh/ivankobzarev/90/orig 2022-03-17T19:07:03.3869324Z * [new branch] gh/ivankobzarev/94/base -> origin/gh/ivankobzarev/94/base 2022-03-17T19:07:03.3870590Z * [new branch] gh/ivankobzarev/94/head -> origin/gh/ivankobzarev/94/head 2022-03-17T19:07:03.3872675Z * [new branch] gh/ivankobzarev/94/orig -> origin/gh/ivankobzarev/94/orig 2022-03-17T19:07:03.3874576Z * [new branch] gh/ivankobzarev/96/base -> origin/gh/ivankobzarev/96/base 2022-03-17T19:07:03.3875774Z * [new branch] gh/ivankobzarev/96/head -> origin/gh/ivankobzarev/96/head 2022-03-17T19:07:03.3877139Z * [new branch] gh/ivankobzarev/96/orig -> origin/gh/ivankobzarev/96/orig 2022-03-17T19:07:03.3879283Z * [new branch] gh/ivankobzarev/97/base -> origin/gh/ivankobzarev/97/base 2022-03-17T19:07:03.3880426Z * [new branch] gh/ivankobzarev/97/head -> origin/gh/ivankobzarev/97/head 2022-03-17T19:07:03.3881914Z * [new branch] gh/ivankobzarev/97/orig -> origin/gh/ivankobzarev/97/orig 2022-03-17T19:07:03.3884020Z * [new branch] gh/ivankobzarev/98/base -> origin/gh/ivankobzarev/98/base 2022-03-17T19:07:03.3885153Z * [new branch] gh/ivankobzarev/98/head -> origin/gh/ivankobzarev/98/head 2022-03-17T19:07:03.3886589Z * [new branch] gh/ivankobzarev/98/orig -> origin/gh/ivankobzarev/98/orig 2022-03-17T19:07:03.3889232Z * [new branch] gh/ivanyashchuk/46/base -> origin/gh/ivanyashchuk/46/base 2022-03-17T19:07:03.3890371Z * [new branch] gh/ivanyashchuk/46/head -> origin/gh/ivanyashchuk/46/head 2022-03-17T19:07:03.3891841Z * [new branch] gh/ivanyashchuk/46/orig -> origin/gh/ivanyashchuk/46/orig 2022-03-17T19:07:03.3894931Z * [new branch] gh/izdeby/31/base -> origin/gh/izdeby/31/base 2022-03-17T19:07:03.3896225Z * [new branch] gh/izdeby/31/head -> origin/gh/izdeby/31/head 2022-03-17T19:07:03.3897734Z * [new branch] gh/izdeby/31/orig -> origin/gh/izdeby/31/orig 2022-03-17T19:07:03.3899264Z * [new branch] gh/izdeby/32/base -> origin/gh/izdeby/32/base 2022-03-17T19:07:03.3901314Z * [new branch] gh/izdeby/32/head -> origin/gh/izdeby/32/head 2022-03-17T19:07:03.3903618Z * [new branch] gh/izdeby/32/orig -> origin/gh/izdeby/32/orig 2022-03-17T19:07:03.3906142Z * [new branch] gh/izdeby/36/base -> origin/gh/izdeby/36/base 2022-03-17T19:07:03.3907507Z * [new branch] gh/izdeby/36/head -> origin/gh/izdeby/36/head 2022-03-17T19:07:03.3908879Z * [new branch] gh/izdeby/36/orig -> origin/gh/izdeby/36/orig 2022-03-17T19:07:03.3911245Z * [new branch] gh/izdeby/39/base -> origin/gh/izdeby/39/base 2022-03-17T19:07:03.3912598Z * [new branch] gh/izdeby/39/head -> origin/gh/izdeby/39/head 2022-03-17T19:07:03.3914412Z * [new branch] gh/izdeby/39/orig -> origin/gh/izdeby/39/orig 2022-03-17T19:07:03.3916458Z * [new branch] gh/izdeby/64/base -> origin/gh/izdeby/64/base 2022-03-17T19:07:03.3917718Z * [new branch] gh/izdeby/64/head -> origin/gh/izdeby/64/head 2022-03-17T19:07:03.3919055Z * [new branch] gh/izdeby/64/orig -> origin/gh/izdeby/64/orig 2022-03-17T19:07:03.3921242Z * [new branch] gh/izdeby/68/base -> origin/gh/izdeby/68/base 2022-03-17T19:07:03.3923413Z * [new branch] gh/izdeby/68/head -> origin/gh/izdeby/68/head 2022-03-17T19:07:03.3924603Z * [new branch] gh/izdeby/68/orig -> origin/gh/izdeby/68/orig 2022-03-17T19:07:03.3927322Z * [new branch] gh/izdeby/69/base -> origin/gh/izdeby/69/base 2022-03-17T19:07:03.3928700Z * [new branch] gh/izdeby/69/head -> origin/gh/izdeby/69/head 2022-03-17T19:07:03.3930067Z * [new branch] gh/izdeby/69/orig -> origin/gh/izdeby/69/orig 2022-03-17T19:07:03.3931898Z * [new branch] gh/izdeby/74/base -> origin/gh/izdeby/74/base 2022-03-17T19:07:03.3933321Z * [new branch] gh/izdeby/74/head -> origin/gh/izdeby/74/head 2022-03-17T19:07:03.3934722Z * [new branch] gh/izdeby/74/orig -> origin/gh/izdeby/74/orig 2022-03-17T19:07:03.3936823Z * [new branch] gh/izdeby/77/base -> origin/gh/izdeby/77/base 2022-03-17T19:07:03.3937928Z * [new branch] gh/izdeby/77/head -> origin/gh/izdeby/77/head 2022-03-17T19:07:03.3939404Z * [new branch] gh/izdeby/77/orig -> origin/gh/izdeby/77/orig 2022-03-17T19:07:03.3941288Z * [new branch] gh/izdeby/79/base -> origin/gh/izdeby/79/base 2022-03-17T19:07:03.3942360Z * [new branch] gh/izdeby/79/head -> origin/gh/izdeby/79/head 2022-03-17T19:07:03.3944369Z * [new branch] gh/izdeby/79/orig -> origin/gh/izdeby/79/orig 2022-03-17T19:07:03.3946860Z * [new branch] gh/izdeby/82/base -> origin/gh/izdeby/82/base 2022-03-17T19:07:03.3948625Z * [new branch] gh/izdeby/82/orig -> origin/gh/izdeby/82/orig 2022-03-17T19:07:03.3949713Z * [new branch] gh/izdeby/87/base -> origin/gh/izdeby/87/base 2022-03-17T19:07:03.3951116Z * [new branch] gh/izdeby/87/head -> origin/gh/izdeby/87/head 2022-03-17T19:07:03.3952522Z * [new branch] gh/izdeby/87/orig -> origin/gh/izdeby/87/orig 2022-03-17T19:07:03.3954007Z * [new branch] gh/izdeby/88/base -> origin/gh/izdeby/88/base 2022-03-17T19:07:03.3955216Z * [new branch] gh/izdeby/88/head -> origin/gh/izdeby/88/head 2022-03-17T19:07:03.3956530Z * [new branch] gh/izdeby/88/orig -> origin/gh/izdeby/88/orig 2022-03-17T19:07:03.3958007Z * [new branch] gh/izdeby/89/base -> origin/gh/izdeby/89/base 2022-03-17T19:07:03.3959466Z * [new branch] gh/izdeby/89/head -> origin/gh/izdeby/89/head 2022-03-17T19:07:03.3960835Z * [new branch] gh/izdeby/89/orig -> origin/gh/izdeby/89/orig 2022-03-17T19:07:03.3963228Z * [new branch] gh/jaceyca/1/base -> origin/gh/jaceyca/1/base 2022-03-17T19:07:03.3964869Z * [new branch] gh/jaceyca/2/base -> origin/gh/jaceyca/2/base 2022-03-17T19:07:03.3966305Z * [new branch] gh/jaceyca/2/head -> origin/gh/jaceyca/2/head 2022-03-17T19:07:03.3967697Z * [new branch] gh/jaceyca/2/orig -> origin/gh/jaceyca/2/orig 2022-03-17T19:07:03.3969804Z * [new branch] gh/jaceyca/3/base -> origin/gh/jaceyca/3/base 2022-03-17T19:07:03.3971001Z * [new branch] gh/jaceyca/3/head -> origin/gh/jaceyca/3/head 2022-03-17T19:07:03.3972484Z * [new branch] gh/jaceyca/3/orig -> origin/gh/jaceyca/3/orig 2022-03-17T19:07:03.3974460Z * [new branch] gh/jaceyca/4/base -> origin/gh/jaceyca/4/base 2022-03-17T19:07:03.3975587Z * [new branch] gh/jaceyca/4/orig -> origin/gh/jaceyca/4/orig 2022-03-17T19:07:03.3977707Z * [new branch] gh/jaceyca/5/base -> origin/gh/jaceyca/5/base 2022-03-17T19:07:03.3979343Z * [new branch] gh/jacobszwejbka/10/base -> origin/gh/jacobszwejbka/10/base 2022-03-17T19:07:03.3981318Z * [new branch] gh/jacobszwejbka/10/head -> origin/gh/jacobszwejbka/10/head 2022-03-17T19:07:03.3982610Z * [new branch] gh/jacobszwejbka/10/orig -> origin/gh/jacobszwejbka/10/orig 2022-03-17T19:07:03.3984840Z * [new branch] gh/jacobszwejbka/19/base -> origin/gh/jacobszwejbka/19/base 2022-03-17T19:07:03.3986051Z * [new branch] gh/jacobszwejbka/19/head -> origin/gh/jacobszwejbka/19/head 2022-03-17T19:07:03.3987568Z * [new branch] gh/jacobszwejbka/19/orig -> origin/gh/jacobszwejbka/19/orig 2022-03-17T19:07:03.3989775Z * [new branch] gh/jacobszwejbka/2/head -> origin/gh/jacobszwejbka/2/head 2022-03-17T19:07:03.3990599Z * [new branch] gh/jacobszwejbka/2/orig -> origin/gh/jacobszwejbka/2/orig 2022-03-17T19:07:03.3992964Z * [new branch] gh/jacobszwejbka/20/base -> origin/gh/jacobszwejbka/20/base 2022-03-17T19:07:03.3994198Z * [new branch] gh/jacobszwejbka/20/head -> origin/gh/jacobszwejbka/20/head 2022-03-17T19:07:03.3995700Z * [new branch] gh/jacobszwejbka/20/orig -> origin/gh/jacobszwejbka/20/orig 2022-03-17T19:07:03.3997930Z * [new branch] gh/jacobszwejbka/24/base -> origin/gh/jacobszwejbka/24/base 2022-03-17T19:07:03.3999151Z * [new branch] gh/jacobszwejbka/24/head -> origin/gh/jacobszwejbka/24/head 2022-03-17T19:07:03.4000707Z * [new branch] gh/jacobszwejbka/24/orig -> origin/gh/jacobszwejbka/24/orig 2022-03-17T19:07:03.4002918Z * [new branch] gh/jacobszwejbka/25/base -> origin/gh/jacobszwejbka/25/base 2022-03-17T19:07:03.4004215Z * [new branch] gh/jacobszwejbka/25/head -> origin/gh/jacobszwejbka/25/head 2022-03-17T19:07:03.4005721Z * [new branch] gh/jacobszwejbka/25/orig -> origin/gh/jacobszwejbka/25/orig 2022-03-17T19:07:03.4007850Z * [new branch] gh/jacobszwejbka/29/base -> origin/gh/jacobszwejbka/29/base 2022-03-17T19:07:03.4009017Z * [new branch] gh/jacobszwejbka/29/head -> origin/gh/jacobszwejbka/29/head 2022-03-17T19:07:03.4010513Z * [new branch] gh/jacobszwejbka/29/orig -> origin/gh/jacobszwejbka/29/orig 2022-03-17T19:07:03.4012679Z * [new branch] gh/jacobszwejbka/31/base -> origin/gh/jacobszwejbka/31/base 2022-03-17T19:07:03.4013911Z * [new branch] gh/jacobszwejbka/31/head -> origin/gh/jacobszwejbka/31/head 2022-03-17T19:07:03.4015495Z * [new branch] gh/jacobszwejbka/32/base -> origin/gh/jacobszwejbka/32/base 2022-03-17T19:07:03.4016969Z * [new branch] gh/jacobszwejbka/32/head -> origin/gh/jacobszwejbka/32/head 2022-03-17T19:07:03.4018508Z * [new branch] gh/jacobszwejbka/33/base -> origin/gh/jacobszwejbka/33/base 2022-03-17T19:07:03.4020694Z * [new branch] gh/jacobszwejbka/35/base -> origin/gh/jacobszwejbka/35/base 2022-03-17T19:07:03.4022094Z * [new branch] gh/jacobszwejbka/35/head -> origin/gh/jacobszwejbka/35/head 2022-03-17T19:07:03.4024422Z * [new branch] gh/jacobszwejbka/36/base -> origin/gh/jacobszwejbka/36/base 2022-03-17T19:07:03.4025699Z * [new branch] gh/jacobszwejbka/36/head -> origin/gh/jacobszwejbka/36/head 2022-03-17T19:07:03.4028048Z * [new branch] gh/jacobszwejbka/36/orig -> origin/gh/jacobszwejbka/36/orig 2022-03-17T19:07:03.4029602Z * [new branch] gh/jacobszwejbka/43/base -> origin/gh/jacobszwejbka/43/base 2022-03-17T19:07:03.4031072Z * [new branch] gh/jacobszwejbka/43/head -> origin/gh/jacobszwejbka/43/head 2022-03-17T19:07:03.4032561Z * [new branch] gh/jacobszwejbka/43/orig -> origin/gh/jacobszwejbka/43/orig 2022-03-17T19:07:03.4034910Z * [new branch] gh/jamesr66a/214/head -> origin/gh/jamesr66a/214/head 2022-03-17T19:07:03.4036139Z * [new branch] gh/jamesr66a/214/orig -> origin/gh/jamesr66a/214/orig 2022-03-17T19:07:03.4038396Z * [new branch] gh/jamesr66a/261/base -> origin/gh/jamesr66a/261/base 2022-03-17T19:07:03.4039520Z * [new branch] gh/jamesr66a/261/head -> origin/gh/jamesr66a/261/head 2022-03-17T19:07:03.4040952Z * [new branch] gh/jamesr66a/261/orig -> origin/gh/jamesr66a/261/orig 2022-03-17T19:07:03.4043385Z * [new branch] gh/jamesr66a/270/base -> origin/gh/jamesr66a/270/base 2022-03-17T19:07:03.4044667Z * [new branch] gh/jamesr66a/270/head -> origin/gh/jamesr66a/270/head 2022-03-17T19:07:03.4046051Z * [new branch] gh/jamesr66a/270/orig -> origin/gh/jamesr66a/270/orig 2022-03-17T19:07:03.4048314Z * [new branch] gh/jamesr66a/272/base -> origin/gh/jamesr66a/272/base 2022-03-17T19:07:03.4049635Z * [new branch] gh/jamesr66a/272/head -> origin/gh/jamesr66a/272/head 2022-03-17T19:07:03.4051013Z * [new branch] gh/jamesr66a/272/orig -> origin/gh/jamesr66a/272/orig 2022-03-17T19:07:03.4053154Z * [new branch] gh/jamesr66a/277/base -> origin/gh/jamesr66a/277/base 2022-03-17T19:07:03.4054233Z * [new branch] gh/jamesr66a/277/head -> origin/gh/jamesr66a/277/head 2022-03-17T19:07:03.4055635Z * [new branch] gh/jamesr66a/277/orig -> origin/gh/jamesr66a/277/orig 2022-03-17T19:07:03.4057260Z * [new branch] gh/jamesr66a/278/base -> origin/gh/jamesr66a/278/base 2022-03-17T19:07:03.4058961Z * [new branch] gh/jamesr66a/278/head -> origin/gh/jamesr66a/278/head 2022-03-17T19:07:03.4060595Z * [new branch] gh/jamesr66a/278/orig -> origin/gh/jamesr66a/278/orig 2022-03-17T19:07:03.4063310Z * [new branch] gh/jamesr66a/280/base -> origin/gh/jamesr66a/280/base 2022-03-17T19:07:03.4064642Z * [new branch] gh/jamesr66a/280/head -> origin/gh/jamesr66a/280/head 2022-03-17T19:07:03.4065995Z * [new branch] gh/jamesr66a/280/orig -> origin/gh/jamesr66a/280/orig 2022-03-17T19:07:03.4068052Z * [new branch] gh/jamesr66a/282/base -> origin/gh/jamesr66a/282/base 2022-03-17T19:07:03.4069283Z * [new branch] gh/jamesr66a/282/head -> origin/gh/jamesr66a/282/head 2022-03-17T19:07:03.4070671Z * [new branch] gh/jamesr66a/282/orig -> origin/gh/jamesr66a/282/orig 2022-03-17T19:07:03.4072735Z * [new branch] gh/jamesr66a/288/base -> origin/gh/jamesr66a/288/base 2022-03-17T19:07:03.4073999Z * [new branch] gh/jamesr66a/288/head -> origin/gh/jamesr66a/288/head 2022-03-17T19:07:03.4075351Z * [new branch] gh/jamesr66a/288/orig -> origin/gh/jamesr66a/288/orig 2022-03-17T19:07:03.4077317Z * [new branch] gh/jamesr66a/311/base -> origin/gh/jamesr66a/311/base 2022-03-17T19:07:03.4078500Z * [new branch] gh/jamesr66a/311/head -> origin/gh/jamesr66a/311/head 2022-03-17T19:07:03.4080113Z * [new branch] gh/jamesr66a/311/orig -> origin/gh/jamesr66a/311/orig 2022-03-17T19:07:03.4082118Z * [new branch] gh/jamesr66a/319/base -> origin/gh/jamesr66a/319/base 2022-03-17T19:07:03.4083228Z * [new branch] gh/jamesr66a/319/head -> origin/gh/jamesr66a/319/head 2022-03-17T19:07:03.4085178Z * [new branch] gh/jamesr66a/319/orig -> origin/gh/jamesr66a/319/orig 2022-03-17T19:07:03.4086865Z * [new branch] gh/jamesr66a/32/base -> origin/gh/jamesr66a/32/base 2022-03-17T19:07:03.4087976Z * [new branch] gh/jamesr66a/32/head -> origin/gh/jamesr66a/32/head 2022-03-17T19:07:03.4089621Z * [new branch] gh/jamesr66a/32/orig -> origin/gh/jamesr66a/32/orig 2022-03-17T19:07:03.4091339Z * [new branch] gh/jamesr66a/322/base -> origin/gh/jamesr66a/322/base 2022-03-17T19:07:03.4093668Z * [new branch] gh/jamesr66a/322/head -> origin/gh/jamesr66a/322/head 2022-03-17T19:07:03.4094875Z * [new branch] gh/jamesr66a/322/orig -> origin/gh/jamesr66a/322/orig 2022-03-17T19:07:03.4097061Z * [new branch] gh/jamesr66a/341/base -> origin/gh/jamesr66a/341/base 2022-03-17T19:07:03.4098113Z * [new branch] gh/jamesr66a/341/head -> origin/gh/jamesr66a/341/head 2022-03-17T19:07:03.4099492Z * [new branch] gh/jamesr66a/341/orig -> origin/gh/jamesr66a/341/orig 2022-03-17T19:07:03.4102643Z * [new branch] gh/jamesr66a/369/base -> origin/gh/jamesr66a/369/base 2022-03-17T19:07:03.4104507Z * [new branch] gh/jamesr66a/369/head -> origin/gh/jamesr66a/369/head 2022-03-17T19:07:03.4105651Z * [new branch] gh/jamesr66a/369/orig -> origin/gh/jamesr66a/369/orig 2022-03-17T19:07:03.4107862Z * [new branch] gh/jamesr66a/370/base -> origin/gh/jamesr66a/370/base 2022-03-17T19:07:03.4108994Z * [new branch] gh/jamesr66a/370/head -> origin/gh/jamesr66a/370/head 2022-03-17T19:07:03.4110504Z * [new branch] gh/jamesr66a/370/orig -> origin/gh/jamesr66a/370/orig 2022-03-17T19:07:03.4111955Z * [new branch] gh/jamesr66a/374/base -> origin/gh/jamesr66a/374/base 2022-03-17T19:07:03.4113809Z * [new branch] gh/jamesr66a/374/head -> origin/gh/jamesr66a/374/head 2022-03-17T19:07:03.4114865Z * [new branch] gh/jamesr66a/374/orig -> origin/gh/jamesr66a/374/orig 2022-03-17T19:07:03.4120418Z * [new branch] gh/jamesr66a/377/base -> origin/gh/jamesr66a/377/base 2022-03-17T19:07:03.4121356Z * [new branch] gh/jamesr66a/377/head -> origin/gh/jamesr66a/377/head 2022-03-17T19:07:03.4121866Z * [new branch] gh/jamesr66a/377/orig -> origin/gh/jamesr66a/377/orig 2022-03-17T19:07:03.4122346Z * [new branch] gh/jamesr66a/399/base -> origin/gh/jamesr66a/399/base 2022-03-17T19:07:03.4122786Z * [new branch] gh/jamesr66a/399/head -> origin/gh/jamesr66a/399/head 2022-03-17T19:07:03.4123812Z * [new branch] gh/jamesr66a/399/orig -> origin/gh/jamesr66a/399/orig 2022-03-17T19:07:03.4125745Z * [new branch] gh/jamesr66a/400/base -> origin/gh/jamesr66a/400/base 2022-03-17T19:07:03.4126908Z * [new branch] gh/jamesr66a/400/head -> origin/gh/jamesr66a/400/head 2022-03-17T19:07:03.4128203Z * [new branch] gh/jamesr66a/400/orig -> origin/gh/jamesr66a/400/orig 2022-03-17T19:07:03.4129982Z * [new branch] gh/jamesr66a/401/base -> origin/gh/jamesr66a/401/base 2022-03-17T19:07:03.4131231Z * [new branch] gh/jamesr66a/401/head -> origin/gh/jamesr66a/401/head 2022-03-17T19:07:03.4132456Z * [new branch] gh/jamesr66a/401/orig -> origin/gh/jamesr66a/401/orig 2022-03-17T19:07:03.4134485Z * [new branch] gh/jamesr66a/403/base -> origin/gh/jamesr66a/403/base 2022-03-17T19:07:03.4135776Z * [new branch] gh/jamesr66a/403/head -> origin/gh/jamesr66a/403/head 2022-03-17T19:07:03.4137928Z * [new branch] gh/jamesr66a/403/orig -> origin/gh/jamesr66a/403/orig 2022-03-17T19:07:03.4139719Z * [new branch] gh/jamesr66a/408/base -> origin/gh/jamesr66a/408/base 2022-03-17T19:07:03.4140968Z * [new branch] gh/jamesr66a/408/head -> origin/gh/jamesr66a/408/head 2022-03-17T19:07:03.4142814Z * [new branch] gh/jamesr66a/408/orig -> origin/gh/jamesr66a/408/orig 2022-03-17T19:07:03.4144541Z * [new branch] gh/jamesr66a/418/base -> origin/gh/jamesr66a/418/base 2022-03-17T19:07:03.4145888Z * [new branch] gh/jamesr66a/418/head -> origin/gh/jamesr66a/418/head 2022-03-17T19:07:03.4147215Z * [new branch] gh/jamesr66a/418/orig -> origin/gh/jamesr66a/418/orig 2022-03-17T19:07:03.4149302Z * [new branch] gh/jamesr66a/420/base -> origin/gh/jamesr66a/420/base 2022-03-17T19:07:03.4151221Z * [new branch] gh/jamesr66a/420/head -> origin/gh/jamesr66a/420/head 2022-03-17T19:07:03.4152377Z * [new branch] gh/jamesr66a/420/orig -> origin/gh/jamesr66a/420/orig 2022-03-17T19:07:03.4154366Z * [new branch] gh/jamesr66a/425/base -> origin/gh/jamesr66a/425/base 2022-03-17T19:07:03.4155418Z * [new branch] gh/jamesr66a/425/head -> origin/gh/jamesr66a/425/head 2022-03-17T19:07:03.4156889Z * [new branch] gh/jamesr66a/425/orig -> origin/gh/jamesr66a/425/orig 2022-03-17T19:07:03.4159086Z * [new branch] gh/jamesr66a/429/base -> origin/gh/jamesr66a/429/base 2022-03-17T19:07:03.4160365Z * [new branch] gh/jamesr66a/429/head -> origin/gh/jamesr66a/429/head 2022-03-17T19:07:03.4161742Z * [new branch] gh/jamesr66a/429/orig -> origin/gh/jamesr66a/429/orig 2022-03-17T19:07:03.4164122Z * [new branch] gh/jamesr66a/434/base -> origin/gh/jamesr66a/434/base 2022-03-17T19:07:03.4165140Z * [new branch] gh/jamesr66a/434/head -> origin/gh/jamesr66a/434/head 2022-03-17T19:07:03.4167026Z * [new branch] gh/jamesr66a/434/orig -> origin/gh/jamesr66a/434/orig 2022-03-17T19:07:03.4168732Z * [new branch] gh/jamesr66a/435/base -> origin/gh/jamesr66a/435/base 2022-03-17T19:07:03.4170144Z * [new branch] gh/jamesr66a/435/head -> origin/gh/jamesr66a/435/head 2022-03-17T19:07:03.4171444Z * [new branch] gh/jamesr66a/435/orig -> origin/gh/jamesr66a/435/orig 2022-03-17T19:07:03.4173601Z * [new branch] gh/jamesr66a/436/base -> origin/gh/jamesr66a/436/base 2022-03-17T19:07:03.4174680Z * [new branch] gh/jamesr66a/436/head -> origin/gh/jamesr66a/436/head 2022-03-17T19:07:03.4175929Z * [new branch] gh/jamesr66a/436/orig -> origin/gh/jamesr66a/436/orig 2022-03-17T19:07:03.4178601Z * [new branch] gh/jamesr66a/51/base -> origin/gh/jamesr66a/51/base 2022-03-17T19:07:03.4179919Z * [new branch] gh/jamesr66a/51/head -> origin/gh/jamesr66a/51/head 2022-03-17T19:07:03.4181969Z * [new branch] gh/jamesr66a/51/orig -> origin/gh/jamesr66a/51/orig 2022-03-17T19:07:03.4183979Z * [new branch] gh/janeyx99/4/head -> origin/gh/janeyx99/4/head 2022-03-17T19:07:03.4185915Z * [new branch] gh/janeyx99/5/base -> origin/gh/janeyx99/5/base 2022-03-17T19:07:03.4187215Z * [new branch] gh/janeyx99/5/head -> origin/gh/janeyx99/5/head 2022-03-17T19:07:03.4188562Z * [new branch] gh/janeyx99/6/base -> origin/gh/janeyx99/6/base 2022-03-17T19:07:03.4190217Z * [new branch] gh/janeyx99/6/head -> origin/gh/janeyx99/6/head 2022-03-17T19:07:03.4192401Z * [new branch] gh/jbschlosser/32/base -> origin/gh/jbschlosser/32/base 2022-03-17T19:07:03.4194214Z * [new branch] gh/jbschlosser/32/head -> origin/gh/jbschlosser/32/head 2022-03-17T19:07:03.4195266Z * [new branch] gh/jbschlosser/32/orig -> origin/gh/jbschlosser/32/orig 2022-03-17T19:07:03.4197771Z * [new branch] gh/jbschlosser/37/base -> origin/gh/jbschlosser/37/base 2022-03-17T19:07:03.4198752Z * [new branch] gh/jbschlosser/37/head -> origin/gh/jbschlosser/37/head 2022-03-17T19:07:03.4200605Z * [new branch] gh/jbschlosser/37/orig -> origin/gh/jbschlosser/37/orig 2022-03-17T19:07:03.4202272Z * [new branch] gh/jbschlosser/38/base -> origin/gh/jbschlosser/38/base 2022-03-17T19:07:03.4203369Z * [new branch] gh/jbschlosser/38/head -> origin/gh/jbschlosser/38/head 2022-03-17T19:07:03.4205341Z * [new branch] gh/jbschlosser/38/orig -> origin/gh/jbschlosser/38/orig 2022-03-17T19:07:03.4207317Z * [new branch] gh/jerryzh168/104/base -> origin/gh/jerryzh168/104/base 2022-03-17T19:07:03.4209343Z * [new branch] gh/jerryzh168/104/head -> origin/gh/jerryzh168/104/head 2022-03-17T19:07:03.4210429Z * [new branch] gh/jerryzh168/104/orig -> origin/gh/jerryzh168/104/orig 2022-03-17T19:07:03.4212504Z * [new branch] gh/jerryzh168/106/base -> origin/gh/jerryzh168/106/base 2022-03-17T19:07:03.4213669Z * [new branch] gh/jerryzh168/106/head -> origin/gh/jerryzh168/106/head 2022-03-17T19:07:03.4215136Z * [new branch] gh/jerryzh168/106/orig -> origin/gh/jerryzh168/106/orig 2022-03-17T19:07:03.4216738Z * [new branch] gh/jerryzh168/107/base -> origin/gh/jerryzh168/107/base 2022-03-17T19:07:03.4218039Z * [new branch] gh/jerryzh168/107/head -> origin/gh/jerryzh168/107/head 2022-03-17T19:07:03.4219649Z * [new branch] gh/jerryzh168/107/orig -> origin/gh/jerryzh168/107/orig 2022-03-17T19:07:03.4221915Z * [new branch] gh/jerryzh168/108/base -> origin/gh/jerryzh168/108/base 2022-03-17T19:07:03.4223042Z * [new branch] gh/jerryzh168/108/head -> origin/gh/jerryzh168/108/head 2022-03-17T19:07:03.4224932Z * [new branch] gh/jerryzh168/108/orig -> origin/gh/jerryzh168/108/orig 2022-03-17T19:07:03.4226817Z * [new branch] gh/jerryzh168/111/base -> origin/gh/jerryzh168/111/base 2022-03-17T19:07:03.4227877Z * [new branch] gh/jerryzh168/111/head -> origin/gh/jerryzh168/111/head 2022-03-17T19:07:03.4229259Z * [new branch] gh/jerryzh168/111/orig -> origin/gh/jerryzh168/111/orig 2022-03-17T19:07:03.4232270Z * [new branch] gh/jerryzh168/116/base -> origin/gh/jerryzh168/116/base 2022-03-17T19:07:03.4233277Z * [new branch] gh/jerryzh168/116/head -> origin/gh/jerryzh168/116/head 2022-03-17T19:07:03.4234935Z * [new branch] gh/jerryzh168/116/orig -> origin/gh/jerryzh168/116/orig 2022-03-17T19:07:03.4236518Z * [new branch] gh/jerryzh168/155/base -> origin/gh/jerryzh168/155/base 2022-03-17T19:07:03.4238149Z * [new branch] gh/jerryzh168/155/head -> origin/gh/jerryzh168/155/head 2022-03-17T19:07:03.4239487Z * [new branch] gh/jerryzh168/155/orig -> origin/gh/jerryzh168/155/orig 2022-03-17T19:07:03.4241726Z * [new branch] gh/jerryzh168/159/base -> origin/gh/jerryzh168/159/base 2022-03-17T19:07:03.4242890Z * [new branch] gh/jerryzh168/159/head -> origin/gh/jerryzh168/159/head 2022-03-17T19:07:03.4244479Z * [new branch] gh/jerryzh168/159/orig -> origin/gh/jerryzh168/159/orig 2022-03-17T19:07:03.4246899Z * [new branch] gh/jerryzh168/191/base -> origin/gh/jerryzh168/191/base 2022-03-17T19:07:03.4247985Z * [new branch] gh/jerryzh168/191/head -> origin/gh/jerryzh168/191/head 2022-03-17T19:07:03.4249305Z * [new branch] gh/jerryzh168/191/orig -> origin/gh/jerryzh168/191/orig 2022-03-17T19:07:03.4251351Z * [new branch] gh/jerryzh168/193/base -> origin/gh/jerryzh168/193/base 2022-03-17T19:07:03.4252553Z * [new branch] gh/jerryzh168/193/head -> origin/gh/jerryzh168/193/head 2022-03-17T19:07:03.4254007Z * [new branch] gh/jerryzh168/193/orig -> origin/gh/jerryzh168/193/orig 2022-03-17T19:07:03.4256051Z * [new branch] gh/jerryzh168/208/base -> origin/gh/jerryzh168/208/base 2022-03-17T19:07:03.4257382Z * [new branch] gh/jerryzh168/208/head -> origin/gh/jerryzh168/208/head 2022-03-17T19:07:03.4259315Z * [new branch] gh/jerryzh168/208/orig -> origin/gh/jerryzh168/208/orig 2022-03-17T19:07:03.4260816Z * [new branch] gh/jerryzh168/221/base -> origin/gh/jerryzh168/221/base 2022-03-17T19:07:03.4262801Z * [new branch] gh/jerryzh168/221/head -> origin/gh/jerryzh168/221/head 2022-03-17T19:07:03.4263945Z * [new branch] gh/jerryzh168/221/orig -> origin/gh/jerryzh168/221/orig 2022-03-17T19:07:03.4266280Z * [new branch] gh/jerryzh168/224/base -> origin/gh/jerryzh168/224/base 2022-03-17T19:07:03.4267440Z * [new branch] gh/jerryzh168/224/head -> origin/gh/jerryzh168/224/head 2022-03-17T19:07:03.4268768Z * [new branch] gh/jerryzh168/224/orig -> origin/gh/jerryzh168/224/orig 2022-03-17T19:07:03.4270927Z * [new branch] gh/jerryzh168/236/base -> origin/gh/jerryzh168/236/base 2022-03-17T19:07:03.4272092Z * [new branch] gh/jerryzh168/236/head -> origin/gh/jerryzh168/236/head 2022-03-17T19:07:03.4273877Z * [new branch] gh/jerryzh168/236/orig -> origin/gh/jerryzh168/236/orig 2022-03-17T19:07:03.4275817Z * [new branch] gh/jerryzh168/238/base -> origin/gh/jerryzh168/238/base 2022-03-17T19:07:03.4276895Z * [new branch] gh/jerryzh168/238/head -> origin/gh/jerryzh168/238/head 2022-03-17T19:07:03.4278566Z * [new branch] gh/jerryzh168/238/orig -> origin/gh/jerryzh168/238/orig 2022-03-17T19:07:03.4280975Z * [new branch] gh/jerryzh168/256/base -> origin/gh/jerryzh168/256/base 2022-03-17T19:07:03.4282064Z * [new branch] gh/jerryzh168/256/head -> origin/gh/jerryzh168/256/head 2022-03-17T19:07:03.4283715Z * [new branch] gh/jerryzh168/256/orig -> origin/gh/jerryzh168/256/orig 2022-03-17T19:07:03.4285573Z * [new branch] gh/jerryzh168/269/base -> origin/gh/jerryzh168/269/base 2022-03-17T19:07:03.4286859Z * [new branch] gh/jerryzh168/269/head -> origin/gh/jerryzh168/269/head 2022-03-17T19:07:03.4288328Z * [new branch] gh/jerryzh168/269/orig -> origin/gh/jerryzh168/269/orig 2022-03-17T19:07:03.4290358Z * [new branch] gh/jerryzh168/28/base -> origin/gh/jerryzh168/28/base 2022-03-17T19:07:03.4291557Z * [new branch] gh/jerryzh168/28/head -> origin/gh/jerryzh168/28/head 2022-03-17T19:07:03.4293334Z * [new branch] gh/jerryzh168/28/orig -> origin/gh/jerryzh168/28/orig 2022-03-17T19:07:03.4294928Z * [new branch] gh/jerryzh168/284/base -> origin/gh/jerryzh168/284/base 2022-03-17T19:07:03.4296227Z * [new branch] gh/jerryzh168/284/head -> origin/gh/jerryzh168/284/head 2022-03-17T19:07:03.4297865Z * [new branch] gh/jerryzh168/284/orig -> origin/gh/jerryzh168/284/orig 2022-03-17T19:07:03.4299909Z * [new branch] gh/jerryzh168/29/base -> origin/gh/jerryzh168/29/base 2022-03-17T19:07:03.4301071Z * [new branch] gh/jerryzh168/29/head -> origin/gh/jerryzh168/29/head 2022-03-17T19:07:03.4317711Z * [new branch] gh/jerryzh168/29/orig -> origin/gh/jerryzh168/29/orig 2022-03-17T19:07:03.4318619Z * [new branch] gh/jerryzh168/30/base -> origin/gh/jerryzh168/30/base 2022-03-17T19:07:03.4319301Z * [new branch] gh/jerryzh168/30/head -> origin/gh/jerryzh168/30/head 2022-03-17T19:07:03.4319954Z * [new branch] gh/jerryzh168/30/orig -> origin/gh/jerryzh168/30/orig 2022-03-17T19:07:03.4320732Z * [new branch] gh/jerryzh168/318/base -> origin/gh/jerryzh168/318/base 2022-03-17T19:07:03.4321230Z * [new branch] gh/jerryzh168/318/head -> origin/gh/jerryzh168/318/head 2022-03-17T19:07:03.4321636Z * [new branch] gh/jerryzh168/318/orig -> origin/gh/jerryzh168/318/orig 2022-03-17T19:07:03.4322367Z * [new branch] gh/jerryzh168/323/base -> origin/gh/jerryzh168/323/base 2022-03-17T19:07:03.4322952Z * [new branch] gh/jerryzh168/323/head -> origin/gh/jerryzh168/323/head 2022-03-17T19:07:03.4323433Z * [new branch] gh/jerryzh168/323/orig -> origin/gh/jerryzh168/323/orig 2022-03-17T19:07:03.4324222Z * [new branch] gh/jerryzh168/337/base -> origin/gh/jerryzh168/337/base 2022-03-17T19:07:03.4324705Z * [new branch] gh/jerryzh168/337/head -> origin/gh/jerryzh168/337/head 2022-03-17T19:07:03.4325506Z * [new branch] gh/jerryzh168/337/orig -> origin/gh/jerryzh168/337/orig 2022-03-17T19:07:03.4326036Z * [new branch] gh/jerryzh168/353/base -> origin/gh/jerryzh168/353/base 2022-03-17T19:07:03.4326442Z * [new branch] gh/jerryzh168/353/head -> origin/gh/jerryzh168/353/head 2022-03-17T19:07:03.4327094Z * [new branch] gh/jerryzh168/353/orig -> origin/gh/jerryzh168/353/orig 2022-03-17T19:07:03.4327866Z * [new branch] gh/jerryzh168/411/base -> origin/gh/jerryzh168/411/base 2022-03-17T19:07:03.4329064Z * [new branch] gh/jerryzh168/411/head -> origin/gh/jerryzh168/411/head 2022-03-17T19:07:03.4330443Z * [new branch] gh/jerryzh168/411/orig -> origin/gh/jerryzh168/411/orig 2022-03-17T19:07:03.4333044Z * [new branch] gh/jerryzh168/412/base -> origin/gh/jerryzh168/412/base 2022-03-17T19:07:03.4334255Z * [new branch] gh/jerryzh168/412/head -> origin/gh/jerryzh168/412/head 2022-03-17T19:07:03.4335539Z * [new branch] gh/jerryzh168/412/orig -> origin/gh/jerryzh168/412/orig 2022-03-17T19:07:03.4337736Z * [new branch] gh/jerryzh168/43/base -> origin/gh/jerryzh168/43/base 2022-03-17T19:07:03.4338899Z * [new branch] gh/jerryzh168/43/head -> origin/gh/jerryzh168/43/head 2022-03-17T19:07:03.4340611Z * [new branch] gh/jerryzh168/43/orig -> origin/gh/jerryzh168/43/orig 2022-03-17T19:07:03.4343021Z * [new branch] gh/jerryzh168/464/base -> origin/gh/jerryzh168/464/base 2022-03-17T19:07:03.4344119Z * [new branch] gh/jerryzh168/464/head -> origin/gh/jerryzh168/464/head 2022-03-17T19:07:03.4345493Z * [new branch] gh/jerryzh168/464/orig -> origin/gh/jerryzh168/464/orig 2022-03-17T19:07:03.4347832Z * [new branch] gh/jerryzh168/47/base -> origin/gh/jerryzh168/47/base 2022-03-17T19:07:03.4349107Z * [new branch] gh/jerryzh168/47/head -> origin/gh/jerryzh168/47/head 2022-03-17T19:07:03.4350628Z * [new branch] gh/jerryzh168/47/orig -> origin/gh/jerryzh168/47/orig 2022-03-17T19:07:03.4353262Z * [new branch] gh/jerryzh168/48/base -> origin/gh/jerryzh168/48/base 2022-03-17T19:07:03.4354589Z * [new branch] gh/jerryzh168/48/head -> origin/gh/jerryzh168/48/head 2022-03-17T19:07:03.4356164Z * [new branch] gh/jerryzh168/48/orig -> origin/gh/jerryzh168/48/orig 2022-03-17T19:07:03.4358198Z * [new branch] gh/jerryzh168/49/base -> origin/gh/jerryzh168/49/base 2022-03-17T19:07:03.4359490Z * [new branch] gh/jerryzh168/49/head -> origin/gh/jerryzh168/49/head 2022-03-17T19:07:03.4361014Z * [new branch] gh/jerryzh168/49/orig -> origin/gh/jerryzh168/49/orig 2022-03-17T19:07:03.4363176Z * [new branch] gh/jerryzh168/52/base -> origin/gh/jerryzh168/52/base 2022-03-17T19:07:03.4364336Z * [new branch] gh/jerryzh168/52/head -> origin/gh/jerryzh168/52/head 2022-03-17T19:07:03.4367020Z * [new branch] gh/jerryzh168/52/orig -> origin/gh/jerryzh168/52/orig 2022-03-17T19:07:03.4367655Z * [new branch] gh/jerryzh168/521/base -> origin/gh/jerryzh168/521/base 2022-03-17T19:07:03.4369997Z * [new branch] gh/jerryzh168/521/head -> origin/gh/jerryzh168/521/head 2022-03-17T19:07:03.4371169Z * [new branch] gh/jerryzh168/521/orig -> origin/gh/jerryzh168/521/orig 2022-03-17T19:07:03.4372650Z * [new branch] gh/jerryzh168/53/base -> origin/gh/jerryzh168/53/base 2022-03-17T19:07:03.4374135Z * [new branch] gh/jerryzh168/53/head -> origin/gh/jerryzh168/53/head 2022-03-17T19:07:03.4375651Z * [new branch] gh/jerryzh168/53/orig -> origin/gh/jerryzh168/53/orig 2022-03-17T19:07:03.4377695Z * [new branch] gh/jerryzh168/54/base -> origin/gh/jerryzh168/54/base 2022-03-17T19:07:03.4379807Z * [new branch] gh/jerryzh168/54/head -> origin/gh/jerryzh168/54/head 2022-03-17T19:07:03.4381298Z * [new branch] gh/jerryzh168/54/orig -> origin/gh/jerryzh168/54/orig 2022-03-17T19:07:03.4382944Z * [new branch] gh/jerryzh168/556/base -> origin/gh/jerryzh168/556/base 2022-03-17T19:07:03.4384412Z * [new branch] gh/jerryzh168/556/head -> origin/gh/jerryzh168/556/head 2022-03-17T19:07:03.4385846Z * [new branch] gh/jerryzh168/556/orig -> origin/gh/jerryzh168/556/orig 2022-03-17T19:07:03.4388030Z * [new branch] gh/jerryzh168/575/base -> origin/gh/jerryzh168/575/base 2022-03-17T19:07:03.4389217Z * [new branch] gh/jerryzh168/575/head -> origin/gh/jerryzh168/575/head 2022-03-17T19:07:03.4390870Z * [new branch] gh/jerryzh168/575/orig -> origin/gh/jerryzh168/575/orig 2022-03-17T19:07:03.4392973Z * [new branch] gh/jerryzh168/576/base -> origin/gh/jerryzh168/576/base 2022-03-17T19:07:03.4394121Z * [new branch] gh/jerryzh168/576/head -> origin/gh/jerryzh168/576/head 2022-03-17T19:07:03.4395709Z * [new branch] gh/jerryzh168/576/orig -> origin/gh/jerryzh168/576/orig 2022-03-17T19:07:03.4397841Z * [new branch] gh/jerryzh168/593/base -> origin/gh/jerryzh168/593/base 2022-03-17T19:07:03.4398840Z * [new branch] gh/jerryzh168/593/head -> origin/gh/jerryzh168/593/head 2022-03-17T19:07:03.4400324Z * [new branch] gh/jerryzh168/593/orig -> origin/gh/jerryzh168/593/orig 2022-03-17T19:07:03.4402552Z * [new branch] gh/jerryzh168/594/base -> origin/gh/jerryzh168/594/base 2022-03-17T19:07:03.4403626Z * [new branch] gh/jerryzh168/594/head -> origin/gh/jerryzh168/594/head 2022-03-17T19:07:03.4404962Z * [new branch] gh/jerryzh168/594/orig -> origin/gh/jerryzh168/594/orig 2022-03-17T19:07:03.4407169Z * [new branch] gh/jerryzh168/631/base -> origin/gh/jerryzh168/631/base 2022-03-17T19:07:03.4408443Z * [new branch] gh/jerryzh168/631/head -> origin/gh/jerryzh168/631/head 2022-03-17T19:07:03.4409867Z * [new branch] gh/jerryzh168/631/orig -> origin/gh/jerryzh168/631/orig 2022-03-17T19:07:03.4412211Z * [new branch] gh/jerryzh168/632/base -> origin/gh/jerryzh168/632/base 2022-03-17T19:07:03.4413585Z * [new branch] gh/jerryzh168/632/head -> origin/gh/jerryzh168/632/head 2022-03-17T19:07:03.4415092Z * [new branch] gh/jerryzh168/632/orig -> origin/gh/jerryzh168/632/orig 2022-03-17T19:07:03.4416782Z * [new branch] gh/jerryzh168/633/base -> origin/gh/jerryzh168/633/base 2022-03-17T19:07:03.4418143Z * [new branch] gh/jerryzh168/633/head -> origin/gh/jerryzh168/633/head 2022-03-17T19:07:03.4419387Z * [new branch] gh/jerryzh168/633/orig -> origin/gh/jerryzh168/633/orig 2022-03-17T19:07:03.4421397Z * [new branch] gh/jerryzh168/634/base -> origin/gh/jerryzh168/634/base 2022-03-17T19:07:03.4422928Z * [new branch] gh/jerryzh168/634/head -> origin/gh/jerryzh168/634/head 2022-03-17T19:07:03.4424396Z * [new branch] gh/jerryzh168/634/orig -> origin/gh/jerryzh168/634/orig 2022-03-17T19:07:03.4426654Z * [new branch] gh/jerryzh168/645/base -> origin/gh/jerryzh168/645/base 2022-03-17T19:07:03.4427690Z * [new branch] gh/jerryzh168/645/head -> origin/gh/jerryzh168/645/head 2022-03-17T19:07:03.4429210Z * [new branch] gh/jerryzh168/645/orig -> origin/gh/jerryzh168/645/orig 2022-03-17T19:07:03.4431428Z * [new branch] gh/jerryzh168/692/base -> origin/gh/jerryzh168/692/base 2022-03-17T19:07:03.4432668Z * [new branch] gh/jerryzh168/692/head -> origin/gh/jerryzh168/692/head 2022-03-17T19:07:03.4434038Z * [new branch] gh/jerryzh168/692/orig -> origin/gh/jerryzh168/692/orig 2022-03-17T19:07:03.4436853Z * [new branch] gh/jerryzh168/70/base -> origin/gh/jerryzh168/70/base 2022-03-17T19:07:03.4438113Z * [new branch] gh/jerryzh168/70/head -> origin/gh/jerryzh168/70/head 2022-03-17T19:07:03.4439470Z * [new branch] gh/jerryzh168/70/orig -> origin/gh/jerryzh168/70/orig 2022-03-17T19:07:03.4441723Z * [new branch] gh/jerryzh168/715/base -> origin/gh/jerryzh168/715/base 2022-03-17T19:07:03.4442660Z * [new branch] gh/jerryzh168/715/head -> origin/gh/jerryzh168/715/head 2022-03-17T19:07:03.4444099Z * [new branch] gh/jerryzh168/715/orig -> origin/gh/jerryzh168/715/orig 2022-03-17T19:07:03.4446315Z * [new branch] gh/jerryzh168/717/base -> origin/gh/jerryzh168/717/base 2022-03-17T19:07:03.4447310Z * [new branch] gh/jerryzh168/717/head -> origin/gh/jerryzh168/717/head 2022-03-17T19:07:03.4448692Z * [new branch] gh/jerryzh168/717/orig -> origin/gh/jerryzh168/717/orig 2022-03-17T19:07:03.4450924Z * [new branch] gh/jerryzh168/718/base -> origin/gh/jerryzh168/718/base 2022-03-17T19:07:03.4452233Z * [new branch] gh/jerryzh168/718/head -> origin/gh/jerryzh168/718/head 2022-03-17T19:07:03.4453643Z * [new branch] gh/jerryzh168/718/orig -> origin/gh/jerryzh168/718/orig 2022-03-17T19:07:03.4455841Z * [new branch] gh/jerryzh168/738/base -> origin/gh/jerryzh168/738/base 2022-03-17T19:07:03.4457041Z * [new branch] gh/jerryzh168/738/head -> origin/gh/jerryzh168/738/head 2022-03-17T19:07:03.4458426Z * [new branch] gh/jerryzh168/738/orig -> origin/gh/jerryzh168/738/orig 2022-03-17T19:07:03.4460756Z * [new branch] gh/jerryzh168/742/base -> origin/gh/jerryzh168/742/base 2022-03-17T19:07:03.4462047Z * [new branch] gh/jerryzh168/742/head -> origin/gh/jerryzh168/742/head 2022-03-17T19:07:03.4463493Z * [new branch] gh/jerryzh168/742/orig -> origin/gh/jerryzh168/742/orig 2022-03-17T19:07:03.4465396Z * [new branch] gh/jerryzh168/743/base -> origin/gh/jerryzh168/743/base 2022-03-17T19:07:03.4466760Z * [new branch] gh/jerryzh168/743/head -> origin/gh/jerryzh168/743/head 2022-03-17T19:07:03.4468191Z * [new branch] gh/jerryzh168/743/orig -> origin/gh/jerryzh168/743/orig 2022-03-17T19:07:03.4470571Z * [new branch] gh/jerryzh168/744/base -> origin/gh/jerryzh168/744/base 2022-03-17T19:07:03.4471705Z * [new branch] gh/jerryzh168/744/head -> origin/gh/jerryzh168/744/head 2022-03-17T19:07:03.4473683Z * [new branch] gh/jerryzh168/744/orig -> origin/gh/jerryzh168/744/orig 2022-03-17T19:07:03.4475272Z * [new branch] gh/jerryzh168/745/base -> origin/gh/jerryzh168/745/base 2022-03-17T19:07:03.4476730Z * [new branch] gh/jerryzh168/745/head -> origin/gh/jerryzh168/745/head 2022-03-17T19:07:03.4478399Z * [new branch] gh/jerryzh168/745/orig -> origin/gh/jerryzh168/745/orig 2022-03-17T19:07:03.4479787Z * [new branch] gh/jerryzh168/746/base -> origin/gh/jerryzh168/746/base 2022-03-17T19:07:03.4481199Z * [new branch] gh/jerryzh168/746/head -> origin/gh/jerryzh168/746/head 2022-03-17T19:07:03.4482593Z * [new branch] gh/jerryzh168/746/orig -> origin/gh/jerryzh168/746/orig 2022-03-17T19:07:03.4484848Z * [new branch] gh/jerryzh168/747/base -> origin/gh/jerryzh168/747/base 2022-03-17T19:07:03.4485950Z * [new branch] gh/jerryzh168/747/head -> origin/gh/jerryzh168/747/head 2022-03-17T19:07:03.4487425Z * [new branch] gh/jerryzh168/747/orig -> origin/gh/jerryzh168/747/orig 2022-03-17T19:07:03.4489644Z * [new branch] gh/jerryzh168/748/base -> origin/gh/jerryzh168/748/base 2022-03-17T19:07:03.4490574Z * [new branch] gh/jerryzh168/748/head -> origin/gh/jerryzh168/748/head 2022-03-17T19:07:03.4491910Z * [new branch] gh/jerryzh168/748/orig -> origin/gh/jerryzh168/748/orig 2022-03-17T19:07:03.4494214Z * [new branch] gh/jerryzh168/75/base -> origin/gh/jerryzh168/75/base 2022-03-17T19:07:03.4495388Z * [new branch] gh/jerryzh168/75/head -> origin/gh/jerryzh168/75/head 2022-03-17T19:07:03.4497707Z * [new branch] gh/jerryzh168/75/orig -> origin/gh/jerryzh168/75/orig 2022-03-17T19:07:03.4499727Z * [new branch] gh/jerryzh168/96/base -> origin/gh/jerryzh168/96/base 2022-03-17T19:07:03.4501244Z * [new branch] gh/jerryzh168/96/head -> origin/gh/jerryzh168/96/head 2022-03-17T19:07:03.4503477Z * [new branch] gh/jerryzh168/96/orig -> origin/gh/jerryzh168/96/orig 2022-03-17T19:07:03.4504893Z * [new branch] gh/jianyuh/15/base -> origin/gh/jianyuh/15/base 2022-03-17T19:07:03.4506442Z * [new branch] gh/jianyuh/15/head -> origin/gh/jianyuh/15/head 2022-03-17T19:07:03.4507885Z * [new branch] gh/jianyuh/15/orig -> origin/gh/jianyuh/15/orig 2022-03-17T19:07:03.4510026Z * [new branch] gh/jianyuh/18/base -> origin/gh/jianyuh/18/base 2022-03-17T19:07:03.4511027Z * [new branch] gh/jianyuh/18/head -> origin/gh/jianyuh/18/head 2022-03-17T19:07:03.4512485Z * [new branch] gh/jianyuh/18/orig -> origin/gh/jianyuh/18/orig 2022-03-17T19:07:03.4514727Z * [new branch] gh/jianyuh/28/base -> origin/gh/jianyuh/28/base 2022-03-17T19:07:03.4515962Z * [new branch] gh/jianyuh/28/head -> origin/gh/jianyuh/28/head 2022-03-17T19:07:03.4517496Z * [new branch] gh/jianyuh/28/orig -> origin/gh/jianyuh/28/orig 2022-03-17T19:07:03.4519208Z * [new branch] gh/jianyuh/3/base -> origin/gh/jianyuh/3/base 2022-03-17T19:07:03.4520589Z * [new branch] gh/jianyuh/3/head -> origin/gh/jianyuh/3/head 2022-03-17T19:07:03.4522030Z * [new branch] gh/jianyuh/3/orig -> origin/gh/jianyuh/3/orig 2022-03-17T19:07:03.4524038Z * [new branch] gh/jianyuh/43/base -> origin/gh/jianyuh/43/base 2022-03-17T19:07:03.4525408Z * [new branch] gh/jianyuh/43/head -> origin/gh/jianyuh/43/head 2022-03-17T19:07:03.4528798Z * [new branch] gh/jianyuh/43/orig -> origin/gh/jianyuh/43/orig 2022-03-17T19:07:03.4530468Z * [new branch] gh/jianyuh/45/base -> origin/gh/jianyuh/45/base 2022-03-17T19:07:03.4532789Z * [new branch] gh/jianyuh/45/head -> origin/gh/jianyuh/45/head 2022-03-17T19:07:03.4533982Z * [new branch] gh/jianyuh/45/orig -> origin/gh/jianyuh/45/orig 2022-03-17T19:07:03.4535698Z * [new branch] gh/jianyuh/46/base -> origin/gh/jianyuh/46/base 2022-03-17T19:07:03.4537068Z * [new branch] gh/jianyuh/46/head -> origin/gh/jianyuh/46/head 2022-03-17T19:07:03.4538504Z * [new branch] gh/jianyuh/46/orig -> origin/gh/jianyuh/46/orig 2022-03-17T19:07:03.4540751Z * [new branch] gh/jianyuh/49/base -> origin/gh/jianyuh/49/base 2022-03-17T19:07:03.4542010Z * [new branch] gh/jianyuh/49/head -> origin/gh/jianyuh/49/head 2022-03-17T19:07:03.4543434Z * [new branch] gh/jianyuh/49/orig -> origin/gh/jianyuh/49/orig 2022-03-17T19:07:03.4545378Z * [new branch] gh/jianyuh/51/base -> origin/gh/jianyuh/51/base 2022-03-17T19:07:03.4546629Z * [new branch] gh/jianyuh/51/head -> origin/gh/jianyuh/51/head 2022-03-17T19:07:03.4547975Z * [new branch] gh/jianyuh/51/orig -> origin/gh/jianyuh/51/orig 2022-03-17T19:07:03.4549902Z * [new branch] gh/jianyuh/57/base -> origin/gh/jianyuh/57/base 2022-03-17T19:07:03.4551169Z * [new branch] gh/jianyuh/57/head -> origin/gh/jianyuh/57/head 2022-03-17T19:07:03.4552584Z * [new branch] gh/jianyuh/57/orig -> origin/gh/jianyuh/57/orig 2022-03-17T19:07:03.4554543Z * [new branch] gh/jianyuh/58/base -> origin/gh/jianyuh/58/base 2022-03-17T19:07:03.4556602Z * [new branch] gh/jianyuh/58/head -> origin/gh/jianyuh/58/head 2022-03-17T19:07:03.4557846Z * [new branch] gh/jianyuh/58/orig -> origin/gh/jianyuh/58/orig 2022-03-17T19:07:03.4560145Z * [new branch] gh/jianyuh/59/base -> origin/gh/jianyuh/59/base 2022-03-17T19:07:03.4561426Z * [new branch] gh/jianyuh/59/head -> origin/gh/jianyuh/59/head 2022-03-17T19:07:03.4563708Z * [new branch] gh/jianyuh/59/orig -> origin/gh/jianyuh/59/orig 2022-03-17T19:07:03.4565183Z * [new branch] gh/jianyuh/62/base -> origin/gh/jianyuh/62/base 2022-03-17T19:07:03.4567105Z * [new branch] gh/jianyuh/62/head -> origin/gh/jianyuh/62/head 2022-03-17T19:07:03.4568444Z * [new branch] gh/jianyuh/62/orig -> origin/gh/jianyuh/62/orig 2022-03-17T19:07:03.4570617Z * [new branch] gh/jianyuh/64/base -> origin/gh/jianyuh/64/base 2022-03-17T19:07:03.4571752Z * [new branch] gh/jianyuh/64/head -> origin/gh/jianyuh/64/head 2022-03-17T19:07:03.4573504Z * [new branch] gh/jianyuh/64/orig -> origin/gh/jianyuh/64/orig 2022-03-17T19:07:03.4575314Z * [new branch] gh/jianyuh/67/base -> origin/gh/jianyuh/67/base 2022-03-17T19:07:03.4576678Z * [new branch] gh/jianyuh/67/head -> origin/gh/jianyuh/67/head 2022-03-17T19:07:03.4578781Z * [new branch] gh/jianyuh/68/base -> origin/gh/jianyuh/68/base 2022-03-17T19:07:03.4579831Z * [new branch] gh/jianyuh/68/head -> origin/gh/jianyuh/68/head 2022-03-17T19:07:03.4581463Z * [new branch] gh/jianyuh/69/base -> origin/gh/jianyuh/69/base 2022-03-17T19:07:03.4582926Z * [new branch] gh/jianyuh/69/head -> origin/gh/jianyuh/69/head 2022-03-17T19:07:03.4584273Z * [new branch] gh/jianyuh/7/base -> origin/gh/jianyuh/7/base 2022-03-17T19:07:03.4586093Z * [new branch] gh/jianyuh/7/head -> origin/gh/jianyuh/7/head 2022-03-17T19:07:03.4587247Z * [new branch] gh/jianyuh/7/orig -> origin/gh/jianyuh/7/orig 2022-03-17T19:07:03.4589594Z * [new branch] gh/jiayisuse/2/base -> origin/gh/jiayisuse/2/base 2022-03-17T19:07:03.4590772Z * [new branch] gh/jiayisuse/2/head -> origin/gh/jiayisuse/2/head 2022-03-17T19:07:03.4592153Z * [new branch] gh/jiayisuse/2/orig -> origin/gh/jiayisuse/2/orig 2022-03-17T19:07:03.4594455Z * [new branch] gh/jiayisuse/3/base -> origin/gh/jiayisuse/3/base 2022-03-17T19:07:03.4595760Z * [new branch] gh/jiayisuse/3/head -> origin/gh/jiayisuse/3/head 2022-03-17T19:07:03.4597166Z * [new branch] gh/jiayisuse/3/orig -> origin/gh/jiayisuse/3/orig 2022-03-17T19:07:03.4599387Z * [new branch] gh/jjlilley/84/base -> origin/gh/jjlilley/84/base 2022-03-17T19:07:03.4600600Z * [new branch] gh/jjlilley/84/head -> origin/gh/jjlilley/84/head 2022-03-17T19:07:03.4602397Z * [new branch] gh/jjlilley/84/orig -> origin/gh/jjlilley/84/orig 2022-03-17T19:07:03.4604370Z * [new branch] gh/jjsjann123/1/base -> origin/gh/jjsjann123/1/base 2022-03-17T19:07:03.4605747Z * [new branch] gh/jjsjann123/1/head -> origin/gh/jjsjann123/1/head 2022-03-17T19:07:03.4607082Z * [new branch] gh/jjsjann123/1/orig -> origin/gh/jjsjann123/1/orig 2022-03-17T19:07:03.4609322Z * [new branch] gh/jshen/10/base -> origin/gh/jshen/10/base 2022-03-17T19:07:03.4610507Z * [new branch] gh/jshen/10/head -> origin/gh/jshen/10/head 2022-03-17T19:07:03.4612490Z * [new branch] gh/jshen/11/base -> origin/gh/jshen/11/base 2022-03-17T19:07:03.4613463Z * [new branch] gh/jshen/11/head -> origin/gh/jshen/11/head 2022-03-17T19:07:03.4615282Z * [new branch] gh/jshen/3/base -> origin/gh/jshen/3/base 2022-03-17T19:07:03.4616499Z * [new branch] gh/jshen/3/head -> origin/gh/jshen/3/head 2022-03-17T19:07:03.4618289Z * [new branch] gh/jshen/3/orig -> origin/gh/jshen/3/orig 2022-03-17T19:07:03.4619679Z * [new branch] gh/jshen/7/base -> origin/gh/jshen/7/base 2022-03-17T19:07:03.4621742Z * [new branch] gh/jshen/7/head -> origin/gh/jshen/7/head 2022-03-17T19:07:03.4622933Z * [new branch] gh/jshen/7/orig -> origin/gh/jshen/7/orig 2022-03-17T19:07:03.4625095Z * [new branch] gh/kazhou/9/base -> origin/gh/kazhou/9/base 2022-03-17T19:07:03.4626246Z * [new branch] gh/kazhou/9/head -> origin/gh/kazhou/9/head 2022-03-17T19:07:03.4628740Z * [new branch] gh/kimishpatel/22/base -> origin/gh/kimishpatel/22/base 2022-03-17T19:07:03.4630085Z * [new branch] gh/kimishpatel/22/head -> origin/gh/kimishpatel/22/head 2022-03-17T19:07:03.4631678Z * [new branch] gh/kimishpatel/23/base -> origin/gh/kimishpatel/23/base 2022-03-17T19:07:03.4632942Z * [new branch] gh/kimishpatel/23/head -> origin/gh/kimishpatel/23/head 2022-03-17T19:07:03.4635024Z * [new branch] gh/kimishpatel/44/base -> origin/gh/kimishpatel/44/base 2022-03-17T19:07:03.4636393Z * [new branch] gh/kimishpatel/44/head -> origin/gh/kimishpatel/44/head 2022-03-17T19:07:03.4637807Z * [new branch] gh/kimishpatel/44/orig -> origin/gh/kimishpatel/44/orig 2022-03-17T19:07:03.4639805Z * [new branch] gh/kimishpatel/46/base -> origin/gh/kimishpatel/46/base 2022-03-17T19:07:03.4641077Z * [new branch] gh/kimishpatel/46/head -> origin/gh/kimishpatel/46/head 2022-03-17T19:07:03.4642669Z * [new branch] gh/kimishpatel/46/orig -> origin/gh/kimishpatel/46/orig 2022-03-17T19:07:03.4644602Z * [new branch] gh/kimishpatel/59/base -> origin/gh/kimishpatel/59/base 2022-03-17T19:07:03.4645772Z * [new branch] gh/kimishpatel/59/head -> origin/gh/kimishpatel/59/head 2022-03-17T19:07:03.4647152Z * [new branch] gh/kimishpatel/59/orig -> origin/gh/kimishpatel/59/orig 2022-03-17T19:07:03.4649357Z * [new branch] gh/kimishpatel/60/base -> origin/gh/kimishpatel/60/base 2022-03-17T19:07:03.4650653Z * [new branch] gh/kimishpatel/60/head -> origin/gh/kimishpatel/60/head 2022-03-17T19:07:03.4652791Z * [new branch] gh/kimishpatel/64/base -> origin/gh/kimishpatel/64/base 2022-03-17T19:07:03.4654187Z * [new branch] gh/kimishpatel/64/head -> origin/gh/kimishpatel/64/head 2022-03-17T19:07:03.4655820Z * [new branch] gh/kimishpatel/65/base -> origin/gh/kimishpatel/65/base 2022-03-17T19:07:03.4657635Z * [new branch] gh/kimishpatel/65/head -> origin/gh/kimishpatel/65/head 2022-03-17T19:07:03.4659428Z * [new branch] gh/kimishpatel/66/base -> origin/gh/kimishpatel/66/base 2022-03-17T19:07:03.4660810Z * [new branch] gh/kimishpatel/66/head -> origin/gh/kimishpatel/66/head 2022-03-17T19:07:03.4663069Z * [new branch] gh/kimishpatel/67/base -> origin/gh/kimishpatel/67/base 2022-03-17T19:07:03.4664211Z * [new branch] gh/kimishpatel/67/head -> origin/gh/kimishpatel/67/head 2022-03-17T19:07:03.4666372Z * [new branch] gh/kimishpatel/70/base -> origin/gh/kimishpatel/70/base 2022-03-17T19:07:03.4667953Z * [new branch] gh/kimishpatel/70/head -> origin/gh/kimishpatel/70/head 2022-03-17T19:07:03.4669818Z * [new branch] gh/kimishpatel/70/orig -> origin/gh/kimishpatel/70/orig 2022-03-17T19:07:03.4672306Z * [new branch] gh/krovatkin/3/base -> origin/gh/krovatkin/3/base 2022-03-17T19:07:03.4673462Z * [new branch] gh/krovatkin/3/head -> origin/gh/krovatkin/3/head 2022-03-17T19:07:03.4675106Z * [new branch] gh/krovatkin/3/orig -> origin/gh/krovatkin/3/orig 2022-03-17T19:07:03.4676752Z * [new branch] gh/krovatkin/4/base -> origin/gh/krovatkin/4/base 2022-03-17T19:07:03.4677954Z * [new branch] gh/krovatkin/4/head -> origin/gh/krovatkin/4/head 2022-03-17T19:07:03.4679530Z * [new branch] gh/krovatkin/4/orig -> origin/gh/krovatkin/4/orig 2022-03-17T19:07:03.4681294Z * [new branch] gh/krovatkin/5/base -> origin/gh/krovatkin/5/base 2022-03-17T19:07:03.4682430Z * [new branch] gh/krovatkin/5/head -> origin/gh/krovatkin/5/head 2022-03-17T19:07:03.4684077Z * [new branch] gh/krovatkin/5/orig -> origin/gh/krovatkin/5/orig 2022-03-17T19:07:03.4685441Z * [new branch] gh/krovatkin/6/base -> origin/gh/krovatkin/6/base 2022-03-17T19:07:03.4686791Z * [new branch] gh/krovatkin/6/head -> origin/gh/krovatkin/6/head 2022-03-17T19:07:03.4688442Z * [new branch] gh/krovatkin/6/orig -> origin/gh/krovatkin/6/orig 2022-03-17T19:07:03.4690148Z * [new branch] gh/krovatkin/7/base -> origin/gh/krovatkin/7/base 2022-03-17T19:07:03.4691222Z * [new branch] gh/krovatkin/7/head -> origin/gh/krovatkin/7/head 2022-03-17T19:07:03.4693682Z * [new branch] gh/krovatkin/8/base -> origin/gh/krovatkin/8/base 2022-03-17T19:07:03.4694946Z * [new branch] gh/krovatkin/8/head -> origin/gh/krovatkin/8/head 2022-03-17T19:07:03.4697233Z * [new branch] gh/linux-jedi/1/base -> origin/gh/linux-jedi/1/base 2022-03-17T19:07:03.4698418Z * [new branch] gh/linux-jedi/1/orig -> origin/gh/linux-jedi/1/orig 2022-03-17T19:07:03.4701330Z * [new branch] gh/lishanYang/1/base -> origin/gh/lishanYang/1/base 2022-03-17T19:07:03.4703563Z * [new branch] gh/lishanYang/1/head -> origin/gh/lishanYang/1/head 2022-03-17T19:07:03.4705443Z * [new branch] gh/lishanYang/2/base -> origin/gh/lishanYang/2/base 2022-03-17T19:07:03.4707612Z * [new branch] gh/lishanYang/2/head -> origin/gh/lishanYang/2/head 2022-03-17T19:07:03.4709333Z * [new branch] gh/lishanYang/2/orig -> origin/gh/lishanYang/2/orig 2022-03-17T19:07:03.4711026Z * [new branch] gh/lishanYang/3/base -> origin/gh/lishanYang/3/base 2022-03-17T19:07:03.4712648Z * [new branch] gh/lishanYang/3/head -> origin/gh/lishanYang/3/head 2022-03-17T19:07:03.4713561Z * [new branch] gh/lishanYang/3/orig -> origin/gh/lishanYang/3/orig 2022-03-17T19:07:03.4715350Z * [new branch] gh/lishanYang/4/base -> origin/gh/lishanYang/4/base 2022-03-17T19:07:03.4716508Z * [new branch] gh/lishanYang/4/head -> origin/gh/lishanYang/4/head 2022-03-17T19:07:03.4718199Z * [new branch] gh/lishanYang/4/orig -> origin/gh/lishanYang/4/orig 2022-03-17T19:07:03.4719987Z * [new branch] gh/ljk53/117/base -> origin/gh/ljk53/117/base 2022-03-17T19:07:03.4721349Z * [new branch] gh/ljk53/117/orig -> origin/gh/ljk53/117/orig 2022-03-17T19:07:03.4723534Z * [new branch] gh/ljk53/123/orig -> origin/gh/ljk53/123/orig 2022-03-17T19:07:03.4725623Z * [new branch] gh/ljk53/135/base -> origin/gh/ljk53/135/base 2022-03-17T19:07:03.4727168Z * [new branch] gh/ljk53/135/orig -> origin/gh/ljk53/135/orig 2022-03-17T19:07:03.4728763Z * [new branch] gh/ljk53/136/base -> origin/gh/ljk53/136/base 2022-03-17T19:07:03.4729846Z * [new branch] gh/ljk53/136/orig -> origin/gh/ljk53/136/orig 2022-03-17T19:07:03.4732080Z * [new branch] gh/ljk53/147/base -> origin/gh/ljk53/147/base 2022-03-17T19:07:03.4733226Z * [new branch] gh/ljk53/147/head -> origin/gh/ljk53/147/head 2022-03-17T19:07:03.4734870Z * [new branch] gh/ljk53/147/orig -> origin/gh/ljk53/147/orig 2022-03-17T19:07:03.4736573Z * [new branch] gh/ljk53/19/base -> origin/gh/ljk53/19/base 2022-03-17T19:07:03.4737939Z * [new branch] gh/ljk53/19/orig -> origin/gh/ljk53/19/orig 2022-03-17T19:07:03.4739819Z * [new branch] gh/ljk53/20/orig -> origin/gh/ljk53/20/orig 2022-03-17T19:07:03.4741957Z * [new branch] gh/ljk53/5/base -> origin/gh/ljk53/5/base 2022-03-17T19:07:03.4743940Z * [new branch] gh/ljk53/54/head -> origin/gh/ljk53/54/head 2022-03-17T19:07:03.4745139Z * [new branch] gh/ljk53/54/orig -> origin/gh/ljk53/54/orig 2022-03-17T19:07:03.4747111Z * [new branch] gh/ljk53/58/head -> origin/gh/ljk53/58/head 2022-03-17T19:07:03.4748353Z * [new branch] gh/ljk53/58/orig -> origin/gh/ljk53/58/orig 2022-03-17T19:07:03.4750689Z * [new branch] gh/lw/102/base -> origin/gh/lw/102/base 2022-03-17T19:07:03.4752500Z * [new branch] gh/lw/102/head -> origin/gh/lw/102/head 2022-03-17T19:07:03.4754201Z * [new branch] gh/lw/102/orig -> origin/gh/lw/102/orig 2022-03-17T19:07:03.4756084Z * [new branch] gh/lw/106/base -> origin/gh/lw/106/base 2022-03-17T19:07:03.4757911Z * [new branch] gh/lw/106/head -> origin/gh/lw/106/head 2022-03-17T19:07:03.4759549Z * [new branch] gh/lw/106/orig -> origin/gh/lw/106/orig 2022-03-17T19:07:03.4761295Z * [new branch] gh/lw/117/base -> origin/gh/lw/117/base 2022-03-17T19:07:03.4762557Z * [new branch] gh/lw/117/head -> origin/gh/lw/117/head 2022-03-17T19:07:03.4764277Z * [new branch] gh/lw/117/orig -> origin/gh/lw/117/orig 2022-03-17T19:07:03.4766208Z * [new branch] gh/lw/153/base -> origin/gh/lw/153/base 2022-03-17T19:07:03.4767554Z * [new branch] gh/lw/153/orig -> origin/gh/lw/153/orig 2022-03-17T19:07:03.4769659Z * [new branch] gh/lw/159/base -> origin/gh/lw/159/base 2022-03-17T19:07:03.4770694Z * [new branch] gh/lw/159/head -> origin/gh/lw/159/head 2022-03-17T19:07:03.4772467Z * [new branch] gh/lw/159/orig -> origin/gh/lw/159/orig 2022-03-17T19:07:03.4774391Z * [new branch] gh/lw/191/base -> origin/gh/lw/191/base 2022-03-17T19:07:03.4775554Z * [new branch] gh/lw/191/head -> origin/gh/lw/191/head 2022-03-17T19:07:03.4777320Z * [new branch] gh/lw/191/orig -> origin/gh/lw/191/orig 2022-03-17T19:07:03.4779255Z * [new branch] gh/lw/219/base -> origin/gh/lw/219/base 2022-03-17T19:07:03.4780463Z * [new branch] gh/lw/219/head -> origin/gh/lw/219/head 2022-03-17T19:07:03.4782322Z * [new branch] gh/lw/219/orig -> origin/gh/lw/219/orig 2022-03-17T19:07:03.4784888Z * [new branch] gh/lw/223/base -> origin/gh/lw/223/base 2022-03-17T19:07:03.4786247Z * [new branch] gh/lw/223/head -> origin/gh/lw/223/head 2022-03-17T19:07:03.4787847Z * [new branch] gh/lw/223/orig -> origin/gh/lw/223/orig 2022-03-17T19:07:03.4789655Z * [new branch] gh/lw/225/base -> origin/gh/lw/225/base 2022-03-17T19:07:03.4790747Z * [new branch] gh/lw/225/head -> origin/gh/lw/225/head 2022-03-17T19:07:03.4792522Z * [new branch] gh/lw/225/orig -> origin/gh/lw/225/orig 2022-03-17T19:07:03.4794243Z * [new branch] gh/lw/71/base -> origin/gh/lw/71/base 2022-03-17T19:07:03.4795603Z * [new branch] gh/lw/71/head -> origin/gh/lw/71/head 2022-03-17T19:07:03.4797177Z * [new branch] gh/lw/71/orig -> origin/gh/lw/71/orig 2022-03-17T19:07:03.4799032Z * [new branch] gh/lw/75/base -> origin/gh/lw/75/base 2022-03-17T19:07:03.4800285Z * [new branch] gh/lw/75/head -> origin/gh/lw/75/head 2022-03-17T19:07:03.4801999Z * [new branch] gh/lw/75/orig -> origin/gh/lw/75/orig 2022-03-17T19:07:03.4803643Z * [new branch] gh/lw/78/base -> origin/gh/lw/78/base 2022-03-17T19:07:03.4805498Z * [new branch] gh/lw/78/head -> origin/gh/lw/78/head 2022-03-17T19:07:03.4807129Z * [new branch] gh/lw/78/orig -> origin/gh/lw/78/orig 2022-03-17T19:07:03.4808485Z * [new branch] gh/lw/79/base -> origin/gh/lw/79/base 2022-03-17T19:07:03.4810196Z * [new branch] gh/lw/79/head -> origin/gh/lw/79/head 2022-03-17T19:07:03.4811559Z * [new branch] gh/lw/79/orig -> origin/gh/lw/79/orig 2022-03-17T19:07:03.4813476Z * [new branch] gh/lw/80/base -> origin/gh/lw/80/base 2022-03-17T19:07:03.4814764Z * [new branch] gh/lw/80/head -> origin/gh/lw/80/head 2022-03-17T19:07:03.4816533Z * [new branch] gh/lw/80/orig -> origin/gh/lw/80/orig 2022-03-17T19:07:03.4817926Z * [new branch] gh/lw/81/base -> origin/gh/lw/81/base 2022-03-17T19:07:03.4819485Z * [new branch] gh/lw/81/head -> origin/gh/lw/81/head 2022-03-17T19:07:03.4821166Z * [new branch] gh/lw/81/orig -> origin/gh/lw/81/orig 2022-03-17T19:07:03.4822638Z * [new branch] gh/lw/82/base -> origin/gh/lw/82/base 2022-03-17T19:07:03.4824257Z * [new branch] gh/lw/82/head -> origin/gh/lw/82/head 2022-03-17T19:07:03.4826157Z * [new branch] gh/lw/82/orig -> origin/gh/lw/82/orig 2022-03-17T19:07:03.4828109Z * [new branch] gh/lyoka/5/base -> origin/gh/lyoka/5/base 2022-03-17T19:07:03.4829306Z * [new branch] gh/lyoka/5/head -> origin/gh/lyoka/5/head 2022-03-17T19:07:03.4830586Z * [new branch] gh/lyoka/5/orig -> origin/gh/lyoka/5/orig 2022-03-17T19:07:03.4832885Z * [new branch] gh/makslevental/1/base -> origin/gh/makslevental/1/base 2022-03-17T19:07:03.4834107Z * [new branch] gh/makslevental/1/head -> origin/gh/makslevental/1/head 2022-03-17T19:07:03.4835484Z * [new branch] gh/makslevental/1/orig -> origin/gh/makslevental/1/orig 2022-03-17T19:07:03.4837685Z * [new branch] gh/makslevental/10/base -> origin/gh/makslevental/10/base 2022-03-17T19:07:03.4838891Z * [new branch] gh/makslevental/10/head -> origin/gh/makslevental/10/head 2022-03-17T19:07:03.4840361Z * [new branch] gh/makslevental/10/orig -> origin/gh/makslevental/10/orig 2022-03-17T19:07:03.4842353Z * [new branch] gh/makslevental/11/base -> origin/gh/makslevental/11/base 2022-03-17T19:07:03.4843683Z * [new branch] gh/makslevental/11/head -> origin/gh/makslevental/11/head 2022-03-17T19:07:03.4845011Z * [new branch] gh/makslevental/11/orig -> origin/gh/makslevental/11/orig 2022-03-17T19:07:03.4847215Z * [new branch] gh/makslevental/13/base -> origin/gh/makslevental/13/base 2022-03-17T19:07:03.4848316Z * [new branch] gh/makslevental/13/head -> origin/gh/makslevental/13/head 2022-03-17T19:07:03.4849682Z * [new branch] gh/makslevental/13/orig -> origin/gh/makslevental/13/orig 2022-03-17T19:07:03.4851318Z * [new branch] gh/makslevental/15/base -> origin/gh/makslevental/15/base 2022-03-17T19:07:03.4852575Z * [new branch] gh/makslevental/15/head -> origin/gh/makslevental/15/head 2022-03-17T19:07:03.4853992Z * [new branch] gh/makslevental/15/orig -> origin/gh/makslevental/15/orig 2022-03-17T19:07:03.4856007Z * [new branch] gh/makslevental/18/base -> origin/gh/makslevental/18/base 2022-03-17T19:07:03.4857202Z * [new branch] gh/makslevental/18/head -> origin/gh/makslevental/18/head 2022-03-17T19:07:03.4858600Z * [new branch] gh/makslevental/18/orig -> origin/gh/makslevental/18/orig 2022-03-17T19:07:03.4861625Z * [new branch] gh/makslevental/19/base -> origin/gh/makslevental/19/base 2022-03-17T19:07:03.4863060Z * [new branch] gh/makslevental/19/head -> origin/gh/makslevental/19/head 2022-03-17T19:07:03.4864994Z * [new branch] gh/makslevental/19/orig -> origin/gh/makslevental/19/orig 2022-03-17T19:07:03.4866801Z * [new branch] gh/makslevental/27/base -> origin/gh/makslevental/27/base 2022-03-17T19:07:03.4867928Z * [new branch] gh/makslevental/27/head -> origin/gh/makslevental/27/head 2022-03-17T19:07:03.4869322Z * [new branch] gh/makslevental/27/orig -> origin/gh/makslevental/27/orig 2022-03-17T19:07:03.4871353Z * [new branch] gh/makslevental/28/base -> origin/gh/makslevental/28/base 2022-03-17T19:07:03.4872683Z * [new branch] gh/makslevental/28/head -> origin/gh/makslevental/28/head 2022-03-17T19:07:03.4874636Z * [new branch] gh/makslevental/28/orig -> origin/gh/makslevental/28/orig 2022-03-17T19:07:03.4876162Z * [new branch] gh/makslevental/29/base -> origin/gh/makslevental/29/base 2022-03-17T19:07:03.4877468Z * [new branch] gh/makslevental/29/head -> origin/gh/makslevental/29/head 2022-03-17T19:07:03.4878896Z * [new branch] gh/makslevental/29/orig -> origin/gh/makslevental/29/orig 2022-03-17T19:07:03.4881022Z * [new branch] gh/makslevental/30/base -> origin/gh/makslevental/30/base 2022-03-17T19:07:03.4882175Z * [new branch] gh/makslevental/30/head -> origin/gh/makslevental/30/head 2022-03-17T19:07:03.4884080Z * [new branch] gh/makslevental/30/orig -> origin/gh/makslevental/30/orig 2022-03-17T19:07:03.4885887Z * [new branch] gh/makslevental/31/base -> origin/gh/makslevental/31/base 2022-03-17T19:07:03.4887030Z * [new branch] gh/makslevental/31/head -> origin/gh/makslevental/31/head 2022-03-17T19:07:03.4888966Z * [new branch] gh/makslevental/31/orig -> origin/gh/makslevental/31/orig 2022-03-17T19:07:03.4890318Z * [new branch] gh/makslevental/35/base -> origin/gh/makslevental/35/base 2022-03-17T19:07:03.4891733Z * [new branch] gh/makslevental/35/head -> origin/gh/makslevental/35/head 2022-03-17T19:07:03.4893116Z * [new branch] gh/makslevental/35/orig -> origin/gh/makslevental/35/orig 2022-03-17T19:07:03.4895271Z * [new branch] gh/makslevental/36/base -> origin/gh/makslevental/36/base 2022-03-17T19:07:03.4896375Z * [new branch] gh/makslevental/36/orig -> origin/gh/makslevental/36/orig 2022-03-17T19:07:03.4898381Z * [new branch] gh/makslevental/38/base -> origin/gh/makslevental/38/base 2022-03-17T19:07:03.4899501Z * [new branch] gh/makslevental/38/head -> origin/gh/makslevental/38/head 2022-03-17T19:07:03.4901179Z * [new branch] gh/makslevental/38/orig -> origin/gh/makslevental/38/orig 2022-03-17T19:07:03.4903866Z * [new branch] gh/makslevental/40/base -> origin/gh/makslevental/40/base 2022-03-17T19:07:03.4905191Z * [new branch] gh/makslevental/40/head -> origin/gh/makslevental/40/head 2022-03-17T19:07:03.4906501Z * [new branch] gh/makslevental/40/orig -> origin/gh/makslevental/40/orig 2022-03-17T19:07:03.4908458Z * [new branch] gh/makslevental/8/base -> origin/gh/makslevental/8/base 2022-03-17T19:07:03.4909554Z * [new branch] gh/makslevental/8/head -> origin/gh/makslevental/8/head 2022-03-17T19:07:03.4911375Z * [new branch] gh/makslevental/8/orig -> origin/gh/makslevental/8/orig 2022-03-17T19:07:03.4912635Z * [new branch] gh/makslevental/9/base -> origin/gh/makslevental/9/base 2022-03-17T19:07:03.4914556Z * [new branch] gh/makslevental/9/head -> origin/gh/makslevental/9/head 2022-03-17T19:07:03.4915581Z * [new branch] gh/makslevental/9/orig -> origin/gh/makslevental/9/orig 2022-03-17T19:07:03.4918243Z * [new branch] gh/mal2147/1/base -> origin/gh/mal2147/1/base 2022-03-17T19:07:03.4919876Z * [new branch] gh/mal2147/3/base -> origin/gh/mal2147/3/base 2022-03-17T19:07:03.4921665Z * [new branch] gh/mal2147/4/base -> origin/gh/mal2147/4/base 2022-03-17T19:07:03.4923398Z * [new branch] gh/mal2147/5/base -> origin/gh/mal2147/5/base 2022-03-17T19:07:03.4924607Z * [new branch] gh/mal2147/6/base -> origin/gh/mal2147/6/base 2022-03-17T19:07:03.4926336Z * [new branch] gh/mal2147/7/base -> origin/gh/mal2147/7/base 2022-03-17T19:07:03.4927740Z * [new branch] gh/mal2147/9/base -> origin/gh/mal2147/9/base 2022-03-17T19:07:03.4930052Z * [new branch] gh/malfet/1/base -> origin/gh/malfet/1/base 2022-03-17T19:07:03.4931136Z * [new branch] gh/malfet/1/head -> origin/gh/malfet/1/head 2022-03-17T19:07:03.4933211Z * [new branch] gh/malfet/10/base -> origin/gh/malfet/10/base 2022-03-17T19:07:03.4934284Z * [new branch] gh/malfet/10/orig -> origin/gh/malfet/10/orig 2022-03-17T19:07:03.4936796Z * [new branch] gh/marjanfariborz/4/base -> origin/gh/marjanfariborz/4/base 2022-03-17T19:07:03.4937979Z * [new branch] gh/marjanfariborz/4/head -> origin/gh/marjanfariborz/4/head 2022-03-17T19:07:03.4939393Z * [new branch] gh/marjanfariborz/4/orig -> origin/gh/marjanfariborz/4/orig 2022-03-17T19:07:03.4941555Z * [new branch] gh/marjanfariborz/5/base -> origin/gh/marjanfariborz/5/base 2022-03-17T19:07:03.4942951Z * [new branch] gh/marjanfariborz/5/head -> origin/gh/marjanfariborz/5/head 2022-03-17T19:07:03.4944649Z * [new branch] gh/marjanfariborz/5/orig -> origin/gh/marjanfariborz/5/orig 2022-03-17T19:07:03.4946699Z * [new branch] gh/mcr229/1/base -> origin/gh/mcr229/1/base 2022-03-17T19:07:03.4947713Z * [new branch] gh/mcr229/1/head -> origin/gh/mcr229/1/head 2022-03-17T19:07:03.4949114Z * [new branch] gh/mcr229/1/orig -> origin/gh/mcr229/1/orig 2022-03-17T19:07:03.4951381Z * [new branch] gh/mehtanirav/2/base -> origin/gh/mehtanirav/2/base 2022-03-17T19:07:03.4952494Z * [new branch] gh/mehtanirav/2/head -> origin/gh/mehtanirav/2/head 2022-03-17T19:07:03.4953917Z * [new branch] gh/mehtanirav/2/orig -> origin/gh/mehtanirav/2/orig 2022-03-17T19:07:03.4956592Z * [new branch] gh/mikaylagawarecki/39/base -> origin/gh/mikaylagawarecki/39/base 2022-03-17T19:07:03.4957732Z * [new branch] gh/mikaylagawarecki/39/head -> origin/gh/mikaylagawarecki/39/head 2022-03-17T19:07:03.4959131Z * [new branch] gh/mikaylagawarecki/39/orig -> origin/gh/mikaylagawarecki/39/orig 2022-03-17T19:07:03.4961138Z * [new branch] gh/mikaylagawarecki/40/base -> origin/gh/mikaylagawarecki/40/base 2022-03-17T19:07:03.4962158Z * [new branch] gh/mikaylagawarecki/40/head -> origin/gh/mikaylagawarecki/40/head 2022-03-17T19:07:03.4963540Z * [new branch] gh/mikaylagawarecki/40/orig -> origin/gh/mikaylagawarecki/40/orig 2022-03-17T19:07:03.4965538Z * [new branch] gh/mikaylagawarecki/41/base -> origin/gh/mikaylagawarecki/41/base 2022-03-17T19:07:03.4966716Z * [new branch] gh/mikaylagawarecki/41/head -> origin/gh/mikaylagawarecki/41/head 2022-03-17T19:07:03.4968066Z * [new branch] gh/mikaylagawarecki/41/orig -> origin/gh/mikaylagawarecki/41/orig 2022-03-17T19:07:03.4970185Z * [new branch] gh/mikaylagawarecki/42/base -> origin/gh/mikaylagawarecki/42/base 2022-03-17T19:07:03.4971238Z * [new branch] gh/mikaylagawarecki/42/head -> origin/gh/mikaylagawarecki/42/head 2022-03-17T19:07:03.4972861Z * [new branch] gh/mikaylagawarecki/42/orig -> origin/gh/mikaylagawarecki/42/orig 2022-03-17T19:07:03.4974627Z * [new branch] gh/mikaylagawarecki/43/base -> origin/gh/mikaylagawarecki/43/base 2022-03-17T19:07:03.4975683Z * [new branch] gh/mikaylagawarecki/43/head -> origin/gh/mikaylagawarecki/43/head 2022-03-17T19:07:03.4977085Z * [new branch] gh/mikaylagawarecki/43/orig -> origin/gh/mikaylagawarecki/43/orig 2022-03-17T19:07:03.4979184Z * [new branch] gh/mikaylagawarecki/44/base -> origin/gh/mikaylagawarecki/44/base 2022-03-17T19:07:03.4980595Z * [new branch] gh/mikaylagawarecki/44/head -> origin/gh/mikaylagawarecki/44/head 2022-03-17T19:07:03.4982537Z * [new branch] gh/mikaylagawarecki/44/orig -> origin/gh/mikaylagawarecki/44/orig 2022-03-17T19:07:03.4984311Z * [new branch] gh/mikaylagawarecki/45/base -> origin/gh/mikaylagawarecki/45/base 2022-03-17T19:07:03.4985504Z * [new branch] gh/mikaylagawarecki/45/head -> origin/gh/mikaylagawarecki/45/head 2022-03-17T19:07:03.4987081Z * [new branch] gh/mikaylagawarecki/45/orig -> origin/gh/mikaylagawarecki/45/orig 2022-03-17T19:07:03.4989129Z * [new branch] gh/mikaylagawarecki/6/base -> origin/gh/mikaylagawarecki/6/base 2022-03-17T19:07:03.4990114Z * [new branch] gh/mikaylagawarecki/6/orig -> origin/gh/mikaylagawarecki/6/orig 2022-03-17T19:07:03.4991907Z * [new branch] gh/mikaylagawarecki/7/base -> origin/gh/mikaylagawarecki/7/base 2022-03-17T19:07:03.4993500Z * [new branch] gh/mikaylagawarecki/7/orig -> origin/gh/mikaylagawarecki/7/orig 2022-03-17T19:07:03.4995729Z * [new branch] gh/mikeiovine/28/base -> origin/gh/mikeiovine/28/base 2022-03-17T19:07:03.4996894Z * [new branch] gh/mikeiovine/28/head -> origin/gh/mikeiovine/28/head 2022-03-17T19:07:03.4998559Z * [new branch] gh/mikeiovine/28/orig -> origin/gh/mikeiovine/28/orig 2022-03-17T19:07:03.5000544Z * [new branch] gh/mikeiovine/32/base -> origin/gh/mikeiovine/32/base 2022-03-17T19:07:03.5001927Z * [new branch] gh/mikeiovine/32/head -> origin/gh/mikeiovine/32/head 2022-03-17T19:07:03.5003317Z * [new branch] gh/mikeiovine/32/orig -> origin/gh/mikeiovine/32/orig 2022-03-17T19:07:03.5005383Z * [new branch] gh/mikeiovine/33/base -> origin/gh/mikeiovine/33/base 2022-03-17T19:07:03.5006558Z * [new branch] gh/mikeiovine/33/head -> origin/gh/mikeiovine/33/head 2022-03-17T19:07:03.5008364Z * [new branch] gh/mikeiovine/33/orig -> origin/gh/mikeiovine/33/orig 2022-03-17T19:07:03.5010022Z * [new branch] gh/mikeiovine/36/base -> origin/gh/mikeiovine/36/base 2022-03-17T19:07:03.5011291Z * [new branch] gh/mikeiovine/36/head -> origin/gh/mikeiovine/36/head 2022-03-17T19:07:03.5013194Z * [new branch] gh/mikeiovine/36/orig -> origin/gh/mikeiovine/36/orig 2022-03-17T19:07:03.5015097Z * [new branch] gh/mikeiovine/37/base -> origin/gh/mikeiovine/37/base 2022-03-17T19:07:03.5016476Z * [new branch] gh/mikeiovine/37/head -> origin/gh/mikeiovine/37/head 2022-03-17T19:07:03.5018217Z * [new branch] gh/mikeiovine/37/orig -> origin/gh/mikeiovine/37/orig 2022-03-17T19:07:03.5019819Z * [new branch] gh/mikeiovine/38/base -> origin/gh/mikeiovine/38/base 2022-03-17T19:07:03.5021027Z * [new branch] gh/mikeiovine/38/head -> origin/gh/mikeiovine/38/head 2022-03-17T19:07:03.5022995Z * [new branch] gh/mikeiovine/38/orig -> origin/gh/mikeiovine/38/orig 2022-03-17T19:07:03.5025146Z * [new branch] gh/mingfeima/1/base -> origin/gh/mingfeima/1/base 2022-03-17T19:07:03.5026097Z * [new branch] gh/mingfeima/1/head -> origin/gh/mingfeima/1/head 2022-03-17T19:07:03.5027706Z * [new branch] gh/mingfeima/1/orig -> origin/gh/mingfeima/1/orig 2022-03-17T19:07:03.5029490Z * [new branch] gh/mingfeima/11/base -> origin/gh/mingfeima/11/base 2022-03-17T19:07:03.5030749Z * [new branch] gh/mingfeima/11/head -> origin/gh/mingfeima/11/head 2022-03-17T19:07:03.5032542Z * [new branch] gh/mingfeima/11/orig -> origin/gh/mingfeima/11/orig 2022-03-17T19:07:03.5034432Z * [new branch] gh/mingfeima/13/base -> origin/gh/mingfeima/13/base 2022-03-17T19:07:03.5035576Z * [new branch] gh/mingfeima/13/head -> origin/gh/mingfeima/13/head 2022-03-17T19:07:03.5037261Z * [new branch] gh/mingfeima/13/orig -> origin/gh/mingfeima/13/orig 2022-03-17T19:07:03.5038989Z * [new branch] gh/mingfeima/14/base -> origin/gh/mingfeima/14/base 2022-03-17T19:07:03.5040138Z * [new branch] gh/mingfeima/14/head -> origin/gh/mingfeima/14/head 2022-03-17T19:07:03.5041875Z * [new branch] gh/mingfeima/14/orig -> origin/gh/mingfeima/14/orig 2022-03-17T19:07:03.5043588Z * [new branch] gh/mingfeima/20/base -> origin/gh/mingfeima/20/base 2022-03-17T19:07:03.5044802Z * [new branch] gh/mingfeima/20/head -> origin/gh/mingfeima/20/head 2022-03-17T19:07:03.5046225Z * [new branch] gh/mingfeima/20/orig -> origin/gh/mingfeima/20/orig 2022-03-17T19:07:03.5048251Z * [new branch] gh/mingfeima/22/base -> origin/gh/mingfeima/22/base 2022-03-17T19:07:03.5049264Z * [new branch] gh/mingfeima/22/head -> origin/gh/mingfeima/22/head 2022-03-17T19:07:03.5050512Z * [new branch] gh/mingfeima/22/orig -> origin/gh/mingfeima/22/orig 2022-03-17T19:07:03.5052766Z * [new branch] gh/mingfeima/26/base -> origin/gh/mingfeima/26/base 2022-03-17T19:07:03.5053954Z * [new branch] gh/mingfeima/26/head -> origin/gh/mingfeima/26/head 2022-03-17T19:07:03.5055330Z * [new branch] gh/mingfeima/26/orig -> origin/gh/mingfeima/26/orig 2022-03-17T19:07:03.5057353Z * [new branch] gh/mingfeima/27/base -> origin/gh/mingfeima/27/base 2022-03-17T19:07:03.5059747Z * [new branch] gh/mingfeima/28/base -> origin/gh/mingfeima/28/base 2022-03-17T19:07:03.5061216Z * [new branch] gh/mingfeima/28/head -> origin/gh/mingfeima/28/head 2022-03-17T19:07:03.5062634Z * [new branch] gh/mingfeima/28/orig -> origin/gh/mingfeima/28/orig 2022-03-17T19:07:03.5064688Z * [new branch] gh/mingfeima/29/base -> origin/gh/mingfeima/29/base 2022-03-17T19:07:03.5065777Z * [new branch] gh/mingfeima/29/head -> origin/gh/mingfeima/29/head 2022-03-17T19:07:03.5067835Z * [new branch] gh/mingfeima/29/orig -> origin/gh/mingfeima/29/orig 2022-03-17T19:07:03.5070187Z * [new branch] gh/mingfeima/3/base -> origin/gh/mingfeima/3/base 2022-03-17T19:07:03.5071438Z * [new branch] gh/mingfeima/3/head -> origin/gh/mingfeima/3/head 2022-03-17T19:07:03.5073311Z * [new branch] gh/mingfeima/3/orig -> origin/gh/mingfeima/3/orig 2022-03-17T19:07:03.5074939Z * [new branch] gh/mingfeima/30/base -> origin/gh/mingfeima/30/base 2022-03-17T19:07:03.5076288Z * [new branch] gh/mingfeima/30/head -> origin/gh/mingfeima/30/head 2022-03-17T19:07:03.5078315Z * [new branch] gh/mingfeima/31/base -> origin/gh/mingfeima/31/base 2022-03-17T19:07:03.5079507Z * [new branch] gh/mingfeima/31/head -> origin/gh/mingfeima/31/head 2022-03-17T19:07:03.5080903Z * [new branch] gh/mingfeima/31/orig -> origin/gh/mingfeima/31/orig 2022-03-17T19:07:03.5082923Z * [new branch] gh/mingfeima/32/base -> origin/gh/mingfeima/32/base 2022-03-17T19:07:03.5083987Z * [new branch] gh/mingfeima/32/head -> origin/gh/mingfeima/32/head 2022-03-17T19:07:03.5085456Z * [new branch] gh/mingfeima/32/orig -> origin/gh/mingfeima/32/orig 2022-03-17T19:07:03.5087545Z * [new branch] gh/mingfeima/33/base -> origin/gh/mingfeima/33/base 2022-03-17T19:07:03.5088835Z * [new branch] gh/mingfeima/33/head -> origin/gh/mingfeima/33/head 2022-03-17T19:07:03.5090167Z * [new branch] gh/mingfeima/33/orig -> origin/gh/mingfeima/33/orig 2022-03-17T19:07:03.5092248Z * [new branch] gh/mingfeima/34/base -> origin/gh/mingfeima/34/base 2022-03-17T19:07:03.5093382Z * [new branch] gh/mingfeima/34/head -> origin/gh/mingfeima/34/head 2022-03-17T19:07:03.5094754Z * [new branch] gh/mingfeima/34/orig -> origin/gh/mingfeima/34/orig 2022-03-17T19:07:03.5096643Z * [new branch] gh/mingfeima/35/base -> origin/gh/mingfeima/35/base 2022-03-17T19:07:03.5097857Z * [new branch] gh/mingfeima/35/head -> origin/gh/mingfeima/35/head 2022-03-17T19:07:03.5099169Z * [new branch] gh/mingfeima/35/orig -> origin/gh/mingfeima/35/orig 2022-03-17T19:07:03.5101196Z * [new branch] gh/mingfeima/36/base -> origin/gh/mingfeima/36/base 2022-03-17T19:07:03.5103506Z * [new branch] gh/mingfeima/36/head -> origin/gh/mingfeima/36/head 2022-03-17T19:07:03.5104573Z * [new branch] gh/mingfeima/36/orig -> origin/gh/mingfeima/36/orig 2022-03-17T19:07:03.5106639Z * [new branch] gh/mingfeima/38/base -> origin/gh/mingfeima/38/base 2022-03-17T19:07:03.5108122Z * [new branch] gh/mingfeima/39/base -> origin/gh/mingfeima/39/base 2022-03-17T19:07:03.5109471Z * [new branch] gh/mingfeima/39/head -> origin/gh/mingfeima/39/head 2022-03-17T19:07:03.5111092Z * [new branch] gh/mingfeima/39/orig -> origin/gh/mingfeima/39/orig 2022-03-17T19:07:03.5113300Z * [new branch] gh/mingfeima/4/base -> origin/gh/mingfeima/4/base 2022-03-17T19:07:03.5114439Z * [new branch] gh/mingfeima/4/head -> origin/gh/mingfeima/4/head 2022-03-17T19:07:03.5116430Z * [new branch] gh/mingfeima/4/orig -> origin/gh/mingfeima/4/orig 2022-03-17T19:07:03.5117393Z * [new branch] gh/mingfeima/40/base -> origin/gh/mingfeima/40/base 2022-03-17T19:07:03.5118820Z * [new branch] gh/mingfeima/40/head -> origin/gh/mingfeima/40/head 2022-03-17T19:07:03.5120183Z * [new branch] gh/mingfeima/40/orig -> origin/gh/mingfeima/40/orig 2022-03-17T19:07:03.5121995Z * [new branch] gh/mingfeima/41/base -> origin/gh/mingfeima/41/base 2022-03-17T19:07:03.5123104Z * [new branch] gh/mingfeima/41/head -> origin/gh/mingfeima/41/head 2022-03-17T19:07:03.5124468Z * [new branch] gh/mingfeima/41/orig -> origin/gh/mingfeima/41/orig 2022-03-17T19:07:03.5126665Z * [new branch] gh/mingfeima/48/base -> origin/gh/mingfeima/48/base 2022-03-17T19:07:03.5127623Z * [new branch] gh/mingfeima/48/head -> origin/gh/mingfeima/48/head 2022-03-17T19:07:03.5128951Z * [new branch] gh/mingfeima/48/orig -> origin/gh/mingfeima/48/orig 2022-03-17T19:07:03.5131199Z * [new branch] gh/mingfeima/49/base -> origin/gh/mingfeima/49/base 2022-03-17T19:07:03.5132391Z * [new branch] gh/mingfeima/49/head -> origin/gh/mingfeima/49/head 2022-03-17T19:07:03.5133810Z * [new branch] gh/mingfeima/49/orig -> origin/gh/mingfeima/49/orig 2022-03-17T19:07:03.5135991Z * [new branch] gh/mingfeima/50/base -> origin/gh/mingfeima/50/base 2022-03-17T19:07:03.5137033Z * [new branch] gh/mingfeima/50/head -> origin/gh/mingfeima/50/head 2022-03-17T19:07:03.5138416Z * [new branch] gh/mingfeima/50/orig -> origin/gh/mingfeima/50/orig 2022-03-17T19:07:03.5140698Z * [new branch] gh/mingfeima/51/base -> origin/gh/mingfeima/51/base 2022-03-17T19:07:03.5142193Z * [new branch] gh/mingfeima/51/head -> origin/gh/mingfeima/51/head 2022-03-17T19:07:03.5143520Z * [new branch] gh/mingfeima/51/orig -> origin/gh/mingfeima/51/orig 2022-03-17T19:07:03.5145481Z * [new branch] gh/mingfeima/52/base -> origin/gh/mingfeima/52/base 2022-03-17T19:07:03.5146518Z * [new branch] gh/mingfeima/52/head -> origin/gh/mingfeima/52/head 2022-03-17T19:07:03.5148197Z * [new branch] gh/mingfeima/52/orig -> origin/gh/mingfeima/52/orig 2022-03-17T19:07:03.5149837Z * [new branch] gh/mingfeima/53/base -> origin/gh/mingfeima/53/base 2022-03-17T19:07:03.5151493Z * [new branch] gh/mingfeima/53/head -> origin/gh/mingfeima/53/head 2022-03-17T19:07:03.5152678Z * [new branch] gh/mingfeima/53/orig -> origin/gh/mingfeima/53/orig 2022-03-17T19:07:03.5154684Z * [new branch] gh/mingfeima/54/base -> origin/gh/mingfeima/54/base 2022-03-17T19:07:03.5155816Z * [new branch] gh/mingfeima/54/head -> origin/gh/mingfeima/54/head 2022-03-17T19:07:03.5157449Z * [new branch] gh/mingfeima/54/orig -> origin/gh/mingfeima/54/orig 2022-03-17T19:07:03.5158634Z * [new branch] gh/mingfeima/55/base -> origin/gh/mingfeima/55/base 2022-03-17T19:07:03.5160339Z * [new branch] gh/mingfeima/55/head -> origin/gh/mingfeima/55/head 2022-03-17T19:07:03.5161395Z * [new branch] gh/mingfeima/55/orig -> origin/gh/mingfeima/55/orig 2022-03-17T19:07:03.5163338Z * [new branch] gh/mingfeima/56/base -> origin/gh/mingfeima/56/base 2022-03-17T19:07:03.5164493Z * [new branch] gh/mingfeima/56/head -> origin/gh/mingfeima/56/head 2022-03-17T19:07:03.5166092Z * [new branch] gh/mingfeima/56/orig -> origin/gh/mingfeima/56/orig 2022-03-17T19:07:03.5167918Z * [new branch] gh/mingfeima/57/base -> origin/gh/mingfeima/57/base 2022-03-17T19:07:03.5169518Z * [new branch] gh/mingfeima/57/head -> origin/gh/mingfeima/57/head 2022-03-17T19:07:03.5170673Z * [new branch] gh/mingfeima/57/orig -> origin/gh/mingfeima/57/orig 2022-03-17T19:07:03.5173228Z * [new branch] gh/mingfeima/58/base -> origin/gh/mingfeima/58/base 2022-03-17T19:07:03.5175226Z * [new branch] gh/mingfeima/58/head -> origin/gh/mingfeima/58/head 2022-03-17T19:07:03.5176405Z * [new branch] gh/mingfeima/58/orig -> origin/gh/mingfeima/58/orig 2022-03-17T19:07:03.5178255Z * [new branch] gh/mingfeima/59/base -> origin/gh/mingfeima/59/base 2022-03-17T19:07:03.5179519Z * [new branch] gh/mingfeima/59/head -> origin/gh/mingfeima/59/head 2022-03-17T19:07:03.5180813Z * [new branch] gh/mingfeima/59/orig -> origin/gh/mingfeima/59/orig 2022-03-17T19:07:03.5183009Z * [new branch] gh/mingfeima/60/base -> origin/gh/mingfeima/60/base 2022-03-17T19:07:03.5184187Z * [new branch] gh/mingfeima/60/head -> origin/gh/mingfeima/60/head 2022-03-17T19:07:03.5186152Z * [new branch] gh/mingfeima/60/orig -> origin/gh/mingfeima/60/orig 2022-03-17T19:07:03.5187976Z * [new branch] gh/mingfeima/61/base -> origin/gh/mingfeima/61/base 2022-03-17T19:07:03.5189156Z * [new branch] gh/mingfeima/61/head -> origin/gh/mingfeima/61/head 2022-03-17T19:07:03.5190785Z * [new branch] gh/mingfeima/61/orig -> origin/gh/mingfeima/61/orig 2022-03-17T19:07:03.5192849Z * [new branch] gh/mingfeima/62/base -> origin/gh/mingfeima/62/base 2022-03-17T19:07:03.5194118Z * [new branch] gh/mingfeima/62/head -> origin/gh/mingfeima/62/head 2022-03-17T19:07:03.5195699Z * [new branch] gh/mingfeima/62/orig -> origin/gh/mingfeima/62/orig 2022-03-17T19:07:03.5197322Z * [new branch] gh/mingfeima/63/base -> origin/gh/mingfeima/63/base 2022-03-17T19:07:03.5198509Z * [new branch] gh/mingfeima/63/head -> origin/gh/mingfeima/63/head 2022-03-17T19:07:03.5200115Z * [new branch] gh/mingfeima/63/orig -> origin/gh/mingfeima/63/orig 2022-03-17T19:07:03.5201787Z * [new branch] gh/mingfeima/64/base -> origin/gh/mingfeima/64/base 2022-03-17T19:07:03.5202948Z * [new branch] gh/mingfeima/64/head -> origin/gh/mingfeima/64/head 2022-03-17T19:07:03.5204470Z * [new branch] gh/mingfeima/64/orig -> origin/gh/mingfeima/64/orig 2022-03-17T19:07:03.5206317Z * [new branch] gh/mingfeima/65/base -> origin/gh/mingfeima/65/base 2022-03-17T19:07:03.5207371Z * [new branch] gh/mingfeima/65/head -> origin/gh/mingfeima/65/head 2022-03-17T19:07:03.5209087Z * [new branch] gh/mingfeima/65/orig -> origin/gh/mingfeima/65/orig 2022-03-17T19:07:03.5211133Z * [new branch] gh/mingfeima/8/base -> origin/gh/mingfeima/8/base 2022-03-17T19:07:03.5212264Z * [new branch] gh/mingfeima/8/head -> origin/gh/mingfeima/8/head 2022-03-17T19:07:03.5213974Z * [new branch] gh/mingfeima/8/orig -> origin/gh/mingfeima/8/orig 2022-03-17T19:07:03.5215926Z * [new branch] gh/mingzhe09088/1/base -> origin/gh/mingzhe09088/1/base 2022-03-17T19:07:03.5217129Z * [new branch] gh/mingzhe09088/1/head -> origin/gh/mingzhe09088/1/head 2022-03-17T19:07:03.5219310Z * [new branch] gh/mingzhe09088/1/orig -> origin/gh/mingzhe09088/1/orig 2022-03-17T19:07:03.5221143Z * [new branch] gh/mingzhe09088/12/base -> origin/gh/mingzhe09088/12/base 2022-03-17T19:07:03.5222784Z * [new branch] gh/mingzhe09088/12/head -> origin/gh/mingzhe09088/12/head 2022-03-17T19:07:03.5223902Z * [new branch] gh/mingzhe09088/12/orig -> origin/gh/mingzhe09088/12/orig 2022-03-17T19:07:03.5225918Z * [new branch] gh/mingzhe09088/14/base -> origin/gh/mingzhe09088/14/base 2022-03-17T19:07:03.5227119Z * [new branch] gh/mingzhe09088/14/head -> origin/gh/mingzhe09088/14/head 2022-03-17T19:07:03.5228734Z * [new branch] gh/mingzhe09088/14/orig -> origin/gh/mingzhe09088/14/orig 2022-03-17T19:07:03.5230446Z * [new branch] gh/mingzhe09088/2/base -> origin/gh/mingzhe09088/2/base 2022-03-17T19:07:03.5231833Z * [new branch] gh/mingzhe09088/2/head -> origin/gh/mingzhe09088/2/head 2022-03-17T19:07:03.5233550Z * [new branch] gh/mingzhe09088/2/orig -> origin/gh/mingzhe09088/2/orig 2022-03-17T19:07:03.5235257Z * [new branch] gh/mingzhe09088/3/base -> origin/gh/mingzhe09088/3/base 2022-03-17T19:07:03.5236531Z * [new branch] gh/mingzhe09088/3/head -> origin/gh/mingzhe09088/3/head 2022-03-17T19:07:03.5238132Z * [new branch] gh/mingzhe09088/3/orig -> origin/gh/mingzhe09088/3/orig 2022-03-17T19:07:03.5239908Z * [new branch] gh/mingzhe09088/4/base -> origin/gh/mingzhe09088/4/base 2022-03-17T19:07:03.5240974Z * [new branch] gh/mingzhe09088/4/orig -> origin/gh/mingzhe09088/4/orig 2022-03-17T19:07:03.5242931Z * [new branch] gh/mingzhe09088/5/base -> origin/gh/mingzhe09088/5/base 2022-03-17T19:07:03.5244005Z * [new branch] gh/mingzhe09088/5/head -> origin/gh/mingzhe09088/5/head 2022-03-17T19:07:03.5245706Z * [new branch] gh/mingzhe09088/5/orig -> origin/gh/mingzhe09088/5/orig 2022-03-17T19:07:03.5247513Z * [new branch] gh/mingzhe09088/6/base -> origin/gh/mingzhe09088/6/base 2022-03-17T19:07:03.5248794Z * [new branch] gh/mingzhe09088/6/head -> origin/gh/mingzhe09088/6/head 2022-03-17T19:07:03.5250665Z * [new branch] gh/mingzhe09088/6/orig -> origin/gh/mingzhe09088/6/orig 2022-03-17T19:07:03.5251924Z * [new branch] gh/mingzhe09088/7/base -> origin/gh/mingzhe09088/7/base 2022-03-17T19:07:03.5253920Z * [new branch] gh/mingzhe09088/7/head -> origin/gh/mingzhe09088/7/head 2022-03-17T19:07:03.5255119Z * [new branch] gh/mingzhe09088/7/orig -> origin/gh/mingzhe09088/7/orig 2022-03-17T19:07:03.5257233Z * [new branch] gh/mingzhe09088/8/base -> origin/gh/mingzhe09088/8/base 2022-03-17T19:07:03.5258387Z * [new branch] gh/mingzhe09088/8/head -> origin/gh/mingzhe09088/8/head 2022-03-17T19:07:03.5260102Z * [new branch] gh/mingzhe09088/8/orig -> origin/gh/mingzhe09088/8/orig 2022-03-17T19:07:03.5262396Z * [new branch] gh/mrshenli/103/base -> origin/gh/mrshenli/103/base 2022-03-17T19:07:03.5263564Z * [new branch] gh/mrshenli/103/head -> origin/gh/mrshenli/103/head 2022-03-17T19:07:03.5265273Z * [new branch] gh/mrshenli/103/orig -> origin/gh/mrshenli/103/orig 2022-03-17T19:07:03.5267032Z * [new branch] gh/mrshenli/138/base -> origin/gh/mrshenli/138/base 2022-03-17T19:07:03.5268389Z * [new branch] gh/mrshenli/138/head -> origin/gh/mrshenli/138/head 2022-03-17T19:07:03.5270236Z * [new branch] gh/mrshenli/138/orig -> origin/gh/mrshenli/138/orig 2022-03-17T19:07:03.5271932Z * [new branch] gh/mrshenli/165/base -> origin/gh/mrshenli/165/base 2022-03-17T19:07:03.5273060Z * [new branch] gh/mrshenli/165/head -> origin/gh/mrshenli/165/head 2022-03-17T19:07:03.5274776Z * [new branch] gh/mrshenli/165/orig -> origin/gh/mrshenli/165/orig 2022-03-17T19:07:03.5276374Z * [new branch] gh/mrshenli/171/base -> origin/gh/mrshenli/171/base 2022-03-17T19:07:03.5278073Z * [new branch] gh/mrshenli/171/head -> origin/gh/mrshenli/171/head 2022-03-17T19:07:03.5279140Z * [new branch] gh/mrshenli/171/orig -> origin/gh/mrshenli/171/orig 2022-03-17T19:07:03.5281074Z * [new branch] gh/mrshenli/172/base -> origin/gh/mrshenli/172/base 2022-03-17T19:07:03.5282272Z * [new branch] gh/mrshenli/172/head -> origin/gh/mrshenli/172/head 2022-03-17T19:07:03.5283622Z * [new branch] gh/mrshenli/172/orig -> origin/gh/mrshenli/172/orig 2022-03-17T19:07:03.5285161Z * [new branch] gh/mrshenli/173/base -> origin/gh/mrshenli/173/base 2022-03-17T19:07:03.5286456Z * [new branch] gh/mrshenli/173/head -> origin/gh/mrshenli/173/head 2022-03-17T19:07:03.5287864Z * [new branch] gh/mrshenli/173/orig -> origin/gh/mrshenli/173/orig 2022-03-17T19:07:03.5289770Z * [new branch] gh/mrshenli/174/base -> origin/gh/mrshenli/174/base 2022-03-17T19:07:03.5290405Z * [new branch] gh/mrshenli/174/head -> origin/gh/mrshenli/174/head 2022-03-17T19:07:03.5292091Z * [new branch] gh/mrshenli/174/orig -> origin/gh/mrshenli/174/orig 2022-03-17T19:07:03.5293844Z * [new branch] gh/mrshenli/180/base -> origin/gh/mrshenli/180/base 2022-03-17T19:07:03.5295220Z * [new branch] gh/mrshenli/180/head -> origin/gh/mrshenli/180/head 2022-03-17T19:07:03.5296823Z * [new branch] gh/mrshenli/180/orig -> origin/gh/mrshenli/180/orig 2022-03-17T19:07:03.5298705Z * [new branch] gh/mrshenli/183/base -> origin/gh/mrshenli/183/base 2022-03-17T19:07:03.5299868Z * [new branch] gh/mrshenli/183/head -> origin/gh/mrshenli/183/head 2022-03-17T19:07:03.5302466Z * [new branch] gh/mrshenli/183/orig -> origin/gh/mrshenli/183/orig 2022-03-17T19:07:03.5304130Z * [new branch] gh/mrshenli/186/base -> origin/gh/mrshenli/186/base 2022-03-17T19:07:03.5305499Z * [new branch] gh/mrshenli/186/head -> origin/gh/mrshenli/186/head 2022-03-17T19:07:03.5307103Z * [new branch] gh/mrshenli/186/orig -> origin/gh/mrshenli/186/orig 2022-03-17T19:07:03.5309498Z * [new branch] gh/mrshenli/195/base -> origin/gh/mrshenli/195/base 2022-03-17T19:07:03.5310615Z * [new branch] gh/mrshenli/195/head -> origin/gh/mrshenli/195/head 2022-03-17T19:07:03.5312317Z * [new branch] gh/mrshenli/195/orig -> origin/gh/mrshenli/195/orig 2022-03-17T19:07:03.5314159Z * [new branch] gh/mrshenli/227/base -> origin/gh/mrshenli/227/base 2022-03-17T19:07:03.5315553Z * [new branch] gh/mrshenli/227/head -> origin/gh/mrshenli/227/head 2022-03-17T19:07:03.5316614Z * [new branch] gh/mrshenli/227/orig -> origin/gh/mrshenli/227/orig 2022-03-17T19:07:03.5318703Z * [new branch] gh/mrshenli/240/base -> origin/gh/mrshenli/240/base 2022-03-17T19:07:03.5319879Z * [new branch] gh/mrshenli/240/head -> origin/gh/mrshenli/240/head 2022-03-17T19:07:03.5321678Z * [new branch] gh/mrshenli/240/orig -> origin/gh/mrshenli/240/orig 2022-03-17T19:07:03.5323322Z * [new branch] gh/mrshenli/257/base -> origin/gh/mrshenli/257/base 2022-03-17T19:07:03.5324578Z * [new branch] gh/mrshenli/257/head -> origin/gh/mrshenli/257/head 2022-03-17T19:07:03.5326359Z * [new branch] gh/mrshenli/257/orig -> origin/gh/mrshenli/257/orig 2022-03-17T19:07:03.5328073Z * [new branch] gh/mrshenli/288/base -> origin/gh/mrshenli/288/base 2022-03-17T19:07:03.5329696Z * [new branch] gh/mrshenli/288/head -> origin/gh/mrshenli/288/head 2022-03-17T19:07:03.5330863Z * [new branch] gh/mrshenli/288/orig -> origin/gh/mrshenli/288/orig 2022-03-17T19:07:03.5333059Z * [new branch] gh/mrshenli/302/base -> origin/gh/mrshenli/302/base 2022-03-17T19:07:03.5334428Z * [new branch] gh/mrshenli/302/head -> origin/gh/mrshenli/302/head 2022-03-17T19:07:03.5335796Z * [new branch] gh/mrshenli/302/orig -> origin/gh/mrshenli/302/orig 2022-03-17T19:07:03.5337708Z * [new branch] gh/mrshenli/306/base -> origin/gh/mrshenli/306/base 2022-03-17T19:07:03.5338739Z * [new branch] gh/mrshenli/306/head -> origin/gh/mrshenli/306/head 2022-03-17T19:07:03.5340243Z * [new branch] gh/mrshenli/306/orig -> origin/gh/mrshenli/306/orig 2022-03-17T19:07:03.5342558Z * [new branch] gh/mrshenli/307/base -> origin/gh/mrshenli/307/base 2022-03-17T19:07:03.5344133Z * [new branch] gh/mrshenli/307/head -> origin/gh/mrshenli/307/head 2022-03-17T19:07:03.5345312Z * [new branch] gh/mrshenli/307/orig -> origin/gh/mrshenli/307/orig 2022-03-17T19:07:03.5347439Z * [new branch] gh/mrshenli/309/base -> origin/gh/mrshenli/309/base 2022-03-17T19:07:03.5348485Z * [new branch] gh/mrshenli/309/head -> origin/gh/mrshenli/309/head 2022-03-17T19:07:03.5350423Z * [new branch] gh/mrshenli/309/orig -> origin/gh/mrshenli/309/orig 2022-03-17T19:07:03.5352108Z * [new branch] gh/mrshenli/310/base -> origin/gh/mrshenli/310/base 2022-03-17T19:07:03.5353195Z * [new branch] gh/mrshenli/310/head -> origin/gh/mrshenli/310/head 2022-03-17T19:07:03.5354811Z * [new branch] gh/mrshenli/310/orig -> origin/gh/mrshenli/310/orig 2022-03-17T19:07:03.5356667Z * [new branch] gh/mrshenli/319/base -> origin/gh/mrshenli/319/base 2022-03-17T19:07:03.5358742Z * [new branch] gh/mrshenli/319/head -> origin/gh/mrshenli/319/head 2022-03-17T19:07:03.5359670Z * [new branch] gh/mrshenli/319/orig -> origin/gh/mrshenli/319/orig 2022-03-17T19:07:03.5361785Z * [new branch] gh/mrshenli/320/base -> origin/gh/mrshenli/320/base 2022-03-17T19:07:03.5363765Z * [new branch] gh/mrshenli/320/head -> origin/gh/mrshenli/320/head 2022-03-17T19:07:03.5364888Z * [new branch] gh/mrshenli/320/orig -> origin/gh/mrshenli/320/orig 2022-03-17T19:07:03.5367089Z * [new branch] gh/mrshenli/322/base -> origin/gh/mrshenli/322/base 2022-03-17T19:07:03.5367989Z * [new branch] gh/mrshenli/322/head -> origin/gh/mrshenli/322/head 2022-03-17T19:07:03.5369718Z * [new branch] gh/mrshenli/322/orig -> origin/gh/mrshenli/322/orig 2022-03-17T19:07:03.5371726Z * [new branch] gh/mrshenli/323/base -> origin/gh/mrshenli/323/base 2022-03-17T19:07:03.5373393Z * [new branch] gh/mrshenli/323/head -> origin/gh/mrshenli/323/head 2022-03-17T19:07:03.5374476Z * [new branch] gh/mrshenli/323/orig -> origin/gh/mrshenli/323/orig 2022-03-17T19:07:03.5376447Z * [new branch] gh/mrshenli/324/base -> origin/gh/mrshenli/324/base 2022-03-17T19:07:03.5377637Z * [new branch] gh/mrshenli/324/head -> origin/gh/mrshenli/324/head 2022-03-17T19:07:03.5379294Z * [new branch] gh/mrshenli/324/orig -> origin/gh/mrshenli/324/orig 2022-03-17T19:07:03.5381112Z * [new branch] gh/mrshenli/326/base -> origin/gh/mrshenli/326/base 2022-03-17T19:07:03.5382568Z * [new branch] gh/mrshenli/326/head -> origin/gh/mrshenli/326/head 2022-03-17T19:07:03.5383900Z * [new branch] gh/mrshenli/326/orig -> origin/gh/mrshenli/326/orig 2022-03-17T19:07:03.5385867Z * [new branch] gh/mrshenli/328/base -> origin/gh/mrshenli/328/base 2022-03-17T19:07:03.5387011Z * [new branch] gh/mrshenli/328/head -> origin/gh/mrshenli/328/head 2022-03-17T19:07:03.5388636Z * [new branch] gh/mrshenli/328/orig -> origin/gh/mrshenli/328/orig 2022-03-17T19:07:03.5390919Z * [new branch] gh/mrshenli/329/base -> origin/gh/mrshenli/329/base 2022-03-17T19:07:03.5393109Z * [new branch] gh/mrshenli/329/head -> origin/gh/mrshenli/329/head 2022-03-17T19:07:03.5394185Z * [new branch] gh/mrshenli/329/orig -> origin/gh/mrshenli/329/orig 2022-03-17T19:07:03.5396298Z * [new branch] gh/mrshenli/334/base -> origin/gh/mrshenli/334/base 2022-03-17T19:07:03.5397483Z * [new branch] gh/mrshenli/334/head -> origin/gh/mrshenli/334/head 2022-03-17T19:07:03.5398799Z * [new branch] gh/mrshenli/334/orig -> origin/gh/mrshenli/334/orig 2022-03-17T19:07:03.5400834Z * [new branch] gh/mrshenli/335/base -> origin/gh/mrshenli/335/base 2022-03-17T19:07:03.5402214Z * [new branch] gh/mrshenli/335/head -> origin/gh/mrshenli/335/head 2022-03-17T19:07:03.5403580Z * [new branch] gh/mrshenli/335/orig -> origin/gh/mrshenli/335/orig 2022-03-17T19:07:03.5405894Z * [new branch] gh/mrshenli/336/base -> origin/gh/mrshenli/336/base 2022-03-17T19:07:03.5407038Z * [new branch] gh/mrshenli/336/head -> origin/gh/mrshenli/336/head 2022-03-17T19:07:03.5408994Z * [new branch] gh/mrshenli/336/orig -> origin/gh/mrshenli/336/orig 2022-03-17T19:07:03.5410822Z * [new branch] gh/mrshenli/38/base -> origin/gh/mrshenli/38/base 2022-03-17T19:07:03.5412520Z * [new branch] gh/mrshenli/38/head -> origin/gh/mrshenli/38/head 2022-03-17T19:07:03.5414094Z * [new branch] gh/mrshenli/38/orig -> origin/gh/mrshenli/38/orig 2022-03-17T19:07:03.5416000Z * [new branch] gh/mrshenli/93/base -> origin/gh/mrshenli/93/base 2022-03-17T19:07:03.5417151Z * [new branch] gh/mrshenli/93/head -> origin/gh/mrshenli/93/head 2022-03-17T19:07:03.5418987Z * [new branch] gh/mrshenli/93/orig -> origin/gh/mrshenli/93/orig 2022-03-17T19:07:03.5420719Z * [new branch] gh/mrshenli/95/head -> origin/gh/mrshenli/95/head 2022-03-17T19:07:03.5422300Z * [new branch] gh/mrshenli/95/orig -> origin/gh/mrshenli/95/orig 2022-03-17T19:07:03.5424464Z * [new branch] gh/mrzzd/11/base -> origin/gh/mrzzd/11/base 2022-03-17T19:07:03.5426465Z * [new branch] gh/mrzzd/11/head -> origin/gh/mrzzd/11/head 2022-03-17T19:07:03.5427583Z * [new branch] gh/mrzzd/11/orig -> origin/gh/mrzzd/11/orig 2022-03-17T19:07:03.5429998Z * [new branch] gh/msaroufim/5/base -> origin/gh/msaroufim/5/base 2022-03-17T19:07:03.5431397Z * [new branch] gh/msaroufim/5/head -> origin/gh/msaroufim/5/head 2022-03-17T19:07:03.5432653Z * [new branch] gh/msaroufim/5/orig -> origin/gh/msaroufim/5/orig 2022-03-17T19:07:03.5434834Z * [new branch] gh/mvsampath/1/base -> origin/gh/mvsampath/1/base 2022-03-17T19:07:03.5435989Z * [new branch] gh/mvsampath/1/head -> origin/gh/mvsampath/1/head 2022-03-17T19:07:03.5437578Z * [new branch] gh/mvsampath/1/orig -> origin/gh/mvsampath/1/orig 2022-03-17T19:07:03.5439633Z * [new branch] gh/nairbv/19/base -> origin/gh/nairbv/19/base 2022-03-17T19:07:03.5441402Z * [new branch] gh/nairbv/19/head -> origin/gh/nairbv/19/head 2022-03-17T19:07:03.5442305Z * [new branch] gh/nairbv/19/orig -> origin/gh/nairbv/19/orig 2022-03-17T19:07:03.5444237Z * [new branch] gh/nairbv/2/head -> origin/gh/nairbv/2/head 2022-03-17T19:07:03.5445508Z * [new branch] gh/nairbv/2/orig -> origin/gh/nairbv/2/orig 2022-03-17T19:07:03.5447375Z * [new branch] gh/nairbv/24/base -> origin/gh/nairbv/24/base 2022-03-17T19:07:03.5449025Z * [new branch] gh/nairbv/24/head -> origin/gh/nairbv/24/head 2022-03-17T19:07:03.5450373Z * [new branch] gh/nairbv/24/orig -> origin/gh/nairbv/24/orig 2022-03-17T19:07:03.5451930Z * [new branch] gh/nairbv/25/base -> origin/gh/nairbv/25/base 2022-03-17T19:07:03.5453014Z * [new branch] gh/nairbv/25/head -> origin/gh/nairbv/25/head 2022-03-17T19:07:03.5454828Z * [new branch] gh/nairbv/25/orig -> origin/gh/nairbv/25/orig 2022-03-17T19:07:03.5456535Z * [new branch] gh/nairbv/3/base -> origin/gh/nairbv/3/base 2022-03-17T19:07:03.5457710Z * [new branch] gh/nairbv/3/orig -> origin/gh/nairbv/3/orig 2022-03-17T19:07:03.5460033Z * [new branch] gh/narang99/3/base -> origin/gh/narang99/3/base 2022-03-17T19:07:03.5461318Z * [new branch] gh/narang99/3/head -> origin/gh/narang99/3/head 2022-03-17T19:07:03.5463093Z * [new branch] gh/narang99/3/orig -> origin/gh/narang99/3/orig 2022-03-17T19:07:03.5464820Z * [new branch] gh/narang99/4/base -> origin/gh/narang99/4/base 2022-03-17T19:07:03.5466177Z * [new branch] gh/narang99/4/head -> origin/gh/narang99/4/head 2022-03-17T19:07:03.5468019Z * [new branch] gh/narang99/4/orig -> origin/gh/narang99/4/orig 2022-03-17T19:07:03.5469299Z * [new branch] gh/narang99/5/base -> origin/gh/narang99/5/base 2022-03-17T19:07:03.5470930Z * [new branch] gh/narang99/5/head -> origin/gh/narang99/5/head 2022-03-17T19:07:03.5472036Z * [new branch] gh/narang99/5/orig -> origin/gh/narang99/5/orig 2022-03-17T19:07:03.5474421Z * [new branch] gh/navahgar/28/base -> origin/gh/navahgar/28/base 2022-03-17T19:07:03.5475598Z * [new branch] gh/navahgar/28/head -> origin/gh/navahgar/28/head 2022-03-17T19:07:03.5477334Z * [new branch] gh/navahgar/28/orig -> origin/gh/navahgar/28/orig 2022-03-17T19:07:03.5479023Z * [new branch] gh/navahgar/29/base -> origin/gh/navahgar/29/base 2022-03-17T19:07:03.5480143Z * [new branch] gh/navahgar/29/head -> origin/gh/navahgar/29/head 2022-03-17T19:07:03.5481747Z * [new branch] gh/navahgar/29/orig -> origin/gh/navahgar/29/orig 2022-03-17T19:07:03.5483465Z * [new branch] gh/navahgar/7/base -> origin/gh/navahgar/7/base 2022-03-17T19:07:03.5484644Z * [new branch] gh/navahgar/7/head -> origin/gh/navahgar/7/head 2022-03-17T19:07:03.5486474Z * [new branch] gh/navahgar/7/orig -> origin/gh/navahgar/7/orig 2022-03-17T19:07:03.5487715Z * [new branch] gh/navahgar/8/base -> origin/gh/navahgar/8/base 2022-03-17T19:07:03.5489342Z * [new branch] gh/navahgar/8/head -> origin/gh/navahgar/8/head 2022-03-17T19:07:03.5490482Z * [new branch] gh/navahgar/8/orig -> origin/gh/navahgar/8/orig 2022-03-17T19:07:03.5492220Z * [new branch] gh/navahgar/9/base -> origin/gh/navahgar/9/base 2022-03-17T19:07:03.5493399Z * [new branch] gh/navahgar/9/head -> origin/gh/navahgar/9/head 2022-03-17T19:07:03.5495006Z * [new branch] gh/navahgar/9/orig -> origin/gh/navahgar/9/orig 2022-03-17T19:07:03.5497008Z * [new branch] gh/ngimel/2/base -> origin/gh/ngimel/2/base 2022-03-17T19:07:03.5499283Z * [new branch] gh/nikithamalgifb/2/base -> origin/gh/nikithamalgifb/2/base 2022-03-17T19:07:03.5500531Z * [new branch] gh/nikithamalgifb/2/head -> origin/gh/nikithamalgifb/2/head 2022-03-17T19:07:03.5503026Z * [new branch] gh/nikithamalgifb/2/orig -> origin/gh/nikithamalgifb/2/orig 2022-03-17T19:07:03.5504953Z * [new branch] gh/nikithamalgifb/21/base -> origin/gh/nikithamalgifb/21/base 2022-03-17T19:07:03.5506235Z * [new branch] gh/nikithamalgifb/21/head -> origin/gh/nikithamalgifb/21/head 2022-03-17T19:07:03.5507902Z * [new branch] gh/nikithamalgifb/21/orig -> origin/gh/nikithamalgifb/21/orig 2022-03-17T19:07:03.5509687Z * [new branch] gh/nikithamalgifb/25/base -> origin/gh/nikithamalgifb/25/base 2022-03-17T19:07:03.5510743Z * [new branch] gh/nikithamalgifb/25/head -> origin/gh/nikithamalgifb/25/head 2022-03-17T19:07:03.5512454Z * [new branch] gh/nikithamalgifb/25/orig -> origin/gh/nikithamalgifb/25/orig 2022-03-17T19:07:03.5514208Z * [new branch] gh/nikithamalgifb/27/base -> origin/gh/nikithamalgifb/27/base 2022-03-17T19:07:03.5515417Z * [new branch] gh/nikithamalgifb/27/head -> origin/gh/nikithamalgifb/27/head 2022-03-17T19:07:03.5516840Z * [new branch] gh/nikithamalgifb/27/orig -> origin/gh/nikithamalgifb/27/orig 2022-03-17T19:07:03.5518797Z * [new branch] gh/nikithamalgifb/29/base -> origin/gh/nikithamalgifb/29/base 2022-03-17T19:07:03.5519960Z * [new branch] gh/nikithamalgifb/29/head -> origin/gh/nikithamalgifb/29/head 2022-03-17T19:07:03.5521563Z * [new branch] gh/nikithamalgifb/29/orig -> origin/gh/nikithamalgifb/29/orig 2022-03-17T19:07:03.5523421Z * [new branch] gh/nikithamalgifb/30/base -> origin/gh/nikithamalgifb/30/base 2022-03-17T19:07:03.5524604Z * [new branch] gh/nikithamalgifb/30/head -> origin/gh/nikithamalgifb/30/head 2022-03-17T19:07:03.5526184Z * [new branch] gh/nikithamalgifb/30/orig -> origin/gh/nikithamalgifb/30/orig 2022-03-17T19:07:03.5527594Z * [new branch] gh/nikithamalgifb/31/base -> origin/gh/nikithamalgifb/31/base 2022-03-17T19:07:03.5529214Z * [new branch] gh/nikithamalgifb/31/head -> origin/gh/nikithamalgifb/31/head 2022-03-17T19:07:03.5530273Z * [new branch] gh/nikithamalgifb/31/orig -> origin/gh/nikithamalgifb/31/orig 2022-03-17T19:07:03.5532754Z * [new branch] gh/nikithamalgifb/32/base -> origin/gh/nikithamalgifb/32/base 2022-03-17T19:07:03.5534042Z * [new branch] gh/nikithamalgifb/32/head -> origin/gh/nikithamalgifb/32/head 2022-03-17T19:07:03.5535639Z * [new branch] gh/nikithamalgifb/32/orig -> origin/gh/nikithamalgifb/32/orig 2022-03-17T19:07:03.5537337Z * [new branch] gh/nikithamalgifb/41/base -> origin/gh/nikithamalgifb/41/base 2022-03-17T19:07:03.5539277Z * [new branch] gh/nikithamalgifb/41/head -> origin/gh/nikithamalgifb/41/head 2022-03-17T19:07:03.5540653Z * [new branch] gh/nikithamalgifb/41/orig -> origin/gh/nikithamalgifb/41/orig 2022-03-17T19:07:03.5542774Z * [new branch] gh/nikithamalgifb/43/base -> origin/gh/nikithamalgifb/43/base 2022-03-17T19:07:03.5544166Z * [new branch] gh/nikithamalgifb/43/head -> origin/gh/nikithamalgifb/43/head 2022-03-17T19:07:03.5545763Z * [new branch] gh/nikithamalgifb/43/orig -> origin/gh/nikithamalgifb/43/orig 2022-03-17T19:07:03.5547558Z * [new branch] gh/nikithamalgifb/44/base -> origin/gh/nikithamalgifb/44/base 2022-03-17T19:07:03.5548734Z * [new branch] gh/nikithamalgifb/44/head -> origin/gh/nikithamalgifb/44/head 2022-03-17T19:07:03.5550354Z * [new branch] gh/nikithamalgifb/44/orig -> origin/gh/nikithamalgifb/44/orig 2022-03-17T19:07:03.5552078Z * [new branch] gh/nikithamalgifb/46/base -> origin/gh/nikithamalgifb/46/base 2022-03-17T19:07:03.5553265Z * [new branch] gh/nikithamalgifb/46/head -> origin/gh/nikithamalgifb/46/head 2022-03-17T19:07:03.5554744Z * [new branch] gh/nikithamalgifb/46/orig -> origin/gh/nikithamalgifb/46/orig 2022-03-17T19:07:03.5556350Z * [new branch] gh/nikithamalgifb/47/base -> origin/gh/nikithamalgifb/47/base 2022-03-17T19:07:03.5557481Z * [new branch] gh/nikithamalgifb/47/head -> origin/gh/nikithamalgifb/47/head 2022-03-17T19:07:03.5559156Z * [new branch] gh/nikithamalgifb/47/orig -> origin/gh/nikithamalgifb/47/orig 2022-03-17T19:07:03.5560869Z * [new branch] gh/nikithamalgifb/5/base -> origin/gh/nikithamalgifb/5/base 2022-03-17T19:07:03.5562610Z * [new branch] gh/nikithamalgifb/5/head -> origin/gh/nikithamalgifb/5/head 2022-03-17T19:07:03.5563758Z * [new branch] gh/nikithamalgifb/5/orig -> origin/gh/nikithamalgifb/5/orig 2022-03-17T19:07:03.5565834Z * [new branch] gh/nikithamalgifb/50/base -> origin/gh/nikithamalgifb/50/base 2022-03-17T19:07:03.5567038Z * [new branch] gh/nikithamalgifb/50/head -> origin/gh/nikithamalgifb/50/head 2022-03-17T19:07:03.5568760Z * [new branch] gh/nikithamalgifb/50/orig -> origin/gh/nikithamalgifb/50/orig 2022-03-17T19:07:03.5570497Z * [new branch] gh/nikithamalgifb/9/base -> origin/gh/nikithamalgifb/9/base 2022-03-17T19:07:03.5571668Z * [new branch] gh/nikithamalgifb/9/head -> origin/gh/nikithamalgifb/9/head 2022-03-17T19:07:03.5573429Z * [new branch] gh/nikithamalgifb/9/orig -> origin/gh/nikithamalgifb/9/orig 2022-03-17T19:07:03.5575378Z * [new branch] gh/nivekt/34/base -> origin/gh/nivekt/34/base 2022-03-17T19:07:03.5576522Z * [new branch] gh/nivekt/34/head -> origin/gh/nivekt/34/head 2022-03-17T19:07:03.5578158Z * [new branch] gh/nivekt/34/orig -> origin/gh/nivekt/34/orig 2022-03-17T19:07:03.5579930Z * [new branch] gh/nivekt/38/base -> origin/gh/nivekt/38/base 2022-03-17T19:07:03.5581025Z * [new branch] gh/nivekt/38/head -> origin/gh/nivekt/38/head 2022-03-17T19:07:03.5583079Z * [new branch] gh/nivekt/38/orig -> origin/gh/nivekt/38/orig 2022-03-17T19:07:03.5584882Z * [new branch] gh/nivekt/41/base -> origin/gh/nivekt/41/base 2022-03-17T19:07:03.5586232Z * [new branch] gh/nivekt/41/head -> origin/gh/nivekt/41/head 2022-03-17T19:07:03.5587574Z * [new branch] gh/nivekt/41/orig -> origin/gh/nivekt/41/orig 2022-03-17T19:07:03.5589425Z * [new branch] gh/nivekt/42/base -> origin/gh/nivekt/42/base 2022-03-17T19:07:03.5593144Z * [new branch] gh/nivekt/42/head -> origin/gh/nivekt/42/head 2022-03-17T19:07:03.5593392Z * [new branch] gh/nivekt/42/orig -> origin/gh/nivekt/42/orig 2022-03-17T19:07:03.5593659Z * [new branch] gh/nivekt/49/base -> origin/gh/nivekt/49/base 2022-03-17T19:07:03.5595580Z * [new branch] gh/nivekt/49/head -> origin/gh/nivekt/49/head 2022-03-17T19:07:03.5596765Z * [new branch] gh/nivekt/49/orig -> origin/gh/nivekt/49/orig 2022-03-17T19:07:03.5599146Z * [new branch] gh/nivekt/52/base -> origin/gh/nivekt/52/base 2022-03-17T19:07:03.5600228Z * [new branch] gh/nivekt/52/head -> origin/gh/nivekt/52/head 2022-03-17T19:07:03.5601872Z * [new branch] gh/nivekt/52/orig -> origin/gh/nivekt/52/orig 2022-03-17T19:07:03.5604005Z * [new branch] gh/nivekt/8/base -> origin/gh/nivekt/8/base 2022-03-17T19:07:03.5605191Z * [new branch] gh/nivekt/8/head -> origin/gh/nivekt/8/head 2022-03-17T19:07:03.5606600Z * [new branch] gh/nivekt/8/orig -> origin/gh/nivekt/8/orig 2022-03-17T19:07:03.5608936Z * [new branch] gh/nkaretnikov/25/base -> origin/gh/nkaretnikov/25/base 2022-03-17T19:07:03.5610090Z * [new branch] gh/nkaretnikov/25/head -> origin/gh/nkaretnikov/25/head 2022-03-17T19:07:03.5611396Z * [new branch] gh/nkaretnikov/25/orig -> origin/gh/nkaretnikov/25/orig 2022-03-17T19:07:03.5613412Z * [new branch] gh/nkaretnikov/28/base -> origin/gh/nkaretnikov/28/base 2022-03-17T19:07:03.5614382Z * [new branch] gh/nkaretnikov/28/head -> origin/gh/nkaretnikov/28/head 2022-03-17T19:07:03.5615717Z * [new branch] gh/nkaretnikov/28/orig -> origin/gh/nkaretnikov/28/orig 2022-03-17T19:07:03.5617695Z * [new branch] gh/nkaretnikov/33/base -> origin/gh/nkaretnikov/33/base 2022-03-17T19:07:03.5618841Z * [new branch] gh/nkaretnikov/33/head -> origin/gh/nkaretnikov/33/head 2022-03-17T19:07:03.5620469Z * [new branch] gh/nkaretnikov/33/orig -> origin/gh/nkaretnikov/33/orig 2022-03-17T19:07:03.5622746Z * [new branch] gh/nkaretnikov/34/base -> origin/gh/nkaretnikov/34/base 2022-03-17T19:07:03.5623830Z * [new branch] gh/nkaretnikov/34/head -> origin/gh/nkaretnikov/34/head 2022-03-17T19:07:03.5625316Z * [new branch] gh/nkaretnikov/34/orig -> origin/gh/nkaretnikov/34/orig 2022-03-17T19:07:03.5627472Z * [new branch] gh/osalpekar/1/base -> origin/gh/osalpekar/1/base 2022-03-17T19:07:03.5628721Z * [new branch] gh/osalpekar/1/head -> origin/gh/osalpekar/1/head 2022-03-17T19:07:03.5630445Z * [new branch] gh/osalpekar/102/base -> origin/gh/osalpekar/102/base 2022-03-17T19:07:03.5631687Z * [new branch] gh/osalpekar/102/head -> origin/gh/osalpekar/102/head 2022-03-17T19:07:03.5633943Z * [new branch] gh/osalpekar/102/orig -> origin/gh/osalpekar/102/orig 2022-03-17T19:07:03.5635634Z * [new branch] gh/osalpekar/107/base -> origin/gh/osalpekar/107/base 2022-03-17T19:07:03.5636876Z * [new branch] gh/osalpekar/107/head -> origin/gh/osalpekar/107/head 2022-03-17T19:07:03.5638796Z * [new branch] gh/osalpekar/2/base -> origin/gh/osalpekar/2/base 2022-03-17T19:07:03.5639795Z * [new branch] gh/osalpekar/2/head -> origin/gh/osalpekar/2/head 2022-03-17T19:07:03.5641676Z * [new branch] gh/osalpekar/3/base -> origin/gh/osalpekar/3/base 2022-03-17T19:07:03.5642634Z * [new branch] gh/osalpekar/3/head -> origin/gh/osalpekar/3/head 2022-03-17T19:07:03.5644783Z * [new branch] gh/osalpekar/4/base -> origin/gh/osalpekar/4/base 2022-03-17T19:07:03.5646106Z * [new branch] gh/osalpekar/4/head -> origin/gh/osalpekar/4/head 2022-03-17T19:07:03.5648854Z * [new branch] gh/osalpekar/40/base -> origin/gh/osalpekar/40/base 2022-03-17T19:07:03.5650037Z * [new branch] gh/osalpekar/40/orig -> origin/gh/osalpekar/40/orig 2022-03-17T19:07:03.5651778Z * [new branch] gh/osalpekar/6/base -> origin/gh/osalpekar/6/base 2022-03-17T19:07:03.5653144Z * [new branch] gh/osalpekar/6/head -> origin/gh/osalpekar/6/head 2022-03-17T19:07:03.5654802Z * [new branch] gh/osalpekar/6/orig -> origin/gh/osalpekar/6/orig 2022-03-17T19:07:03.5656785Z * [new branch] gh/osalpekar/60/base -> origin/gh/osalpekar/60/base 2022-03-17T19:07:03.5658017Z * [new branch] gh/osalpekar/60/head -> origin/gh/osalpekar/60/head 2022-03-17T19:07:03.5659956Z * [new branch] gh/osalpekar/60/orig -> origin/gh/osalpekar/60/orig 2022-03-17T19:07:03.5661162Z * [new branch] gh/osalpekar/61/base -> origin/gh/osalpekar/61/base 2022-03-17T19:07:03.5663334Z * [new branch] gh/osalpekar/61/head -> origin/gh/osalpekar/61/head 2022-03-17T19:07:03.5664604Z * [new branch] gh/osalpekar/61/orig -> origin/gh/osalpekar/61/orig 2022-03-17T19:07:03.5666238Z * [new branch] gh/osalpekar/62/base -> origin/gh/osalpekar/62/base 2022-03-17T19:07:03.5676568Z * [new branch] gh/osalpekar/62/head -> origin/gh/osalpekar/62/head 2022-03-17T19:07:03.5677729Z * [new branch] gh/osalpekar/62/orig -> origin/gh/osalpekar/62/orig 2022-03-17T19:07:03.5679389Z * [new branch] gh/osalpekar/76/base -> origin/gh/osalpekar/76/base 2022-03-17T19:07:03.5680860Z * [new branch] gh/osalpekar/76/head -> origin/gh/osalpekar/76/head 2022-03-17T19:07:03.5682468Z * [new branch] gh/osalpekar/76/orig -> origin/gh/osalpekar/76/orig 2022-03-17T19:07:03.5684441Z * [new branch] gh/osalpekar/77/base -> origin/gh/osalpekar/77/base 2022-03-17T19:07:03.5685670Z * [new branch] gh/osalpekar/77/head -> origin/gh/osalpekar/77/head 2022-03-17T19:07:03.5687089Z * [new branch] gh/osalpekar/77/orig -> origin/gh/osalpekar/77/orig 2022-03-17T19:07:03.5689128Z * [new branch] gh/osalpekar/78/base -> origin/gh/osalpekar/78/base 2022-03-17T19:07:03.5690325Z * [new branch] gh/osalpekar/78/head -> origin/gh/osalpekar/78/head 2022-03-17T19:07:03.5691908Z * [new branch] gh/osalpekar/78/orig -> origin/gh/osalpekar/78/orig 2022-03-17T19:07:03.5693882Z * [new branch] gh/osalpekar/85/base -> origin/gh/osalpekar/85/base 2022-03-17T19:07:03.5699492Z * [new branch] gh/osalpekar/85/head -> origin/gh/osalpekar/85/head 2022-03-17T19:07:03.5701176Z * [new branch] gh/osalpekar/85/orig -> origin/gh/osalpekar/85/orig 2022-03-17T19:07:03.5703551Z * [new branch] gh/osalpekar/86/base -> origin/gh/osalpekar/86/base 2022-03-17T19:07:03.5704710Z * [new branch] gh/osalpekar/86/head -> origin/gh/osalpekar/86/head 2022-03-17T19:07:03.5707443Z * [new branch] gh/osalpekar/86/orig -> origin/gh/osalpekar/86/orig 2022-03-17T19:07:03.5710766Z * [new branch] gh/osalpekar/91/base -> origin/gh/osalpekar/91/base 2022-03-17T19:07:03.5712030Z * [new branch] gh/osalpekar/91/head -> origin/gh/osalpekar/91/head 2022-03-17T19:07:03.5713477Z * [new branch] gh/osalpekar/91/orig -> origin/gh/osalpekar/91/orig 2022-03-17T19:07:03.5718092Z * [new branch] gh/pavithranrao/1/base -> origin/gh/pavithranrao/1/base 2022-03-17T19:07:03.5720238Z * [new branch] gh/pavithranrao/1/head -> origin/gh/pavithranrao/1/head 2022-03-17T19:07:03.5721544Z * [new branch] gh/pavithranrao/10/base -> origin/gh/pavithranrao/10/base 2022-03-17T19:07:03.5722885Z * [new branch] gh/pavithranrao/10/head -> origin/gh/pavithranrao/10/head 2022-03-17T19:07:03.5727707Z * [new branch] gh/pavithranrao/11/base -> origin/gh/pavithranrao/11/base 2022-03-17T19:07:03.5728826Z * [new branch] gh/pavithranrao/11/head -> origin/gh/pavithranrao/11/head 2022-03-17T19:07:03.5733091Z * [new branch] gh/pavithranrao/17/base -> origin/gh/pavithranrao/17/base 2022-03-17T19:07:03.5734339Z * [new branch] gh/pavithranrao/17/head -> origin/gh/pavithranrao/17/head 2022-03-17T19:07:03.5737356Z * [new branch] gh/pavithranrao/17/orig -> origin/gh/pavithranrao/17/orig 2022-03-17T19:07:03.5739851Z * [new branch] gh/pavithranrao/19/base -> origin/gh/pavithranrao/19/base 2022-03-17T19:07:03.5741118Z * [new branch] gh/pavithranrao/19/head -> origin/gh/pavithranrao/19/head 2022-03-17T19:07:03.5745341Z * [new branch] gh/pavithranrao/19/orig -> origin/gh/pavithranrao/19/orig 2022-03-17T19:07:03.5747162Z * [new branch] gh/pavithranrao/2/base -> origin/gh/pavithranrao/2/base 2022-03-17T19:07:03.5749313Z * [new branch] gh/pavithranrao/2/head -> origin/gh/pavithranrao/2/head 2022-03-17T19:07:03.5750568Z * [new branch] gh/pavithranrao/20/base -> origin/gh/pavithranrao/20/base 2022-03-17T19:07:03.5752078Z * [new branch] gh/pavithranrao/20/head -> origin/gh/pavithranrao/20/head 2022-03-17T19:07:03.5753442Z * [new branch] gh/pavithranrao/20/orig -> origin/gh/pavithranrao/20/orig 2022-03-17T19:07:03.5755468Z * [new branch] gh/pavithranrao/21/base -> origin/gh/pavithranrao/21/base 2022-03-17T19:07:03.5756592Z * [new branch] gh/pavithranrao/21/head -> origin/gh/pavithranrao/21/head 2022-03-17T19:07:03.5758111Z * [new branch] gh/pavithranrao/21/orig -> origin/gh/pavithranrao/21/orig 2022-03-17T19:07:03.5760137Z * [new branch] gh/pavithranrao/25/base -> origin/gh/pavithranrao/25/base 2022-03-17T19:07:03.5761128Z * [new branch] gh/pavithranrao/25/head -> origin/gh/pavithranrao/25/head 2022-03-17T19:07:03.5763348Z * [new branch] gh/pavithranrao/25/orig -> origin/gh/pavithranrao/25/orig 2022-03-17T19:07:03.5764977Z * [new branch] gh/pavithranrao/26/base -> origin/gh/pavithranrao/26/base 2022-03-17T19:07:03.5766662Z * [new branch] gh/pavithranrao/26/head -> origin/gh/pavithranrao/26/head 2022-03-17T19:07:03.5768728Z * [new branch] gh/pavithranrao/28/base -> origin/gh/pavithranrao/28/base 2022-03-17T19:07:03.5769881Z * [new branch] gh/pavithranrao/28/head -> origin/gh/pavithranrao/28/head 2022-03-17T19:07:03.5771310Z * [new branch] gh/pavithranrao/29/base -> origin/gh/pavithranrao/29/base 2022-03-17T19:07:03.5772638Z * [new branch] gh/pavithranrao/29/head -> origin/gh/pavithranrao/29/head 2022-03-17T19:07:03.5774380Z * [new branch] gh/pavithranrao/3/base -> origin/gh/pavithranrao/3/base 2022-03-17T19:07:03.5775552Z * [new branch] gh/pavithranrao/3/head -> origin/gh/pavithranrao/3/head 2022-03-17T19:07:03.5777323Z * [new branch] gh/pavithranrao/30/base -> origin/gh/pavithranrao/30/base 2022-03-17T19:07:03.5778486Z * [new branch] gh/pavithranrao/30/head -> origin/gh/pavithranrao/30/head 2022-03-17T19:07:03.5779989Z * [new branch] gh/pavithranrao/31/base -> origin/gh/pavithranrao/31/base 2022-03-17T19:07:03.5781351Z * [new branch] gh/pavithranrao/31/head -> origin/gh/pavithranrao/31/head 2022-03-17T19:07:03.5783672Z * [new branch] gh/pavithranrao/32/base -> origin/gh/pavithranrao/32/base 2022-03-17T19:07:03.5784673Z * [new branch] gh/pavithranrao/32/head -> origin/gh/pavithranrao/32/head 2022-03-17T19:07:03.5786361Z * [new branch] gh/pavithranrao/33/base -> origin/gh/pavithranrao/33/base 2022-03-17T19:07:03.5787683Z * [new branch] gh/pavithranrao/33/head -> origin/gh/pavithranrao/33/head 2022-03-17T19:07:03.5789125Z * [new branch] gh/pavithranrao/33/orig -> origin/gh/pavithranrao/33/orig 2022-03-17T19:07:03.5791262Z * [new branch] gh/pavithranrao/34/base -> origin/gh/pavithranrao/34/base 2022-03-17T19:07:03.5792394Z * [new branch] gh/pavithranrao/34/head -> origin/gh/pavithranrao/34/head 2022-03-17T19:07:03.5793809Z * [new branch] gh/pavithranrao/34/orig -> origin/gh/pavithranrao/34/orig 2022-03-17T19:07:03.5795756Z * [new branch] gh/pavithranrao/36/base -> origin/gh/pavithranrao/36/base 2022-03-17T19:07:03.5796932Z * [new branch] gh/pavithranrao/36/head -> origin/gh/pavithranrao/36/head 2022-03-17T19:07:03.5798417Z * [new branch] gh/pavithranrao/36/orig -> origin/gh/pavithranrao/36/orig 2022-03-17T19:07:03.5800522Z * [new branch] gh/pavithranrao/37/base -> origin/gh/pavithranrao/37/base 2022-03-17T19:07:03.5801693Z * [new branch] gh/pavithranrao/37/head -> origin/gh/pavithranrao/37/head 2022-03-17T19:07:03.5803677Z * [new branch] gh/pavithranrao/37/orig -> origin/gh/pavithranrao/37/orig 2022-03-17T19:07:03.5804896Z * [new branch] gh/pavithranrao/39/base -> origin/gh/pavithranrao/39/base 2022-03-17T19:07:03.5806454Z * [new branch] gh/pavithranrao/39/head -> origin/gh/pavithranrao/39/head 2022-03-17T19:07:03.5807753Z * [new branch] gh/pavithranrao/39/orig -> origin/gh/pavithranrao/39/orig 2022-03-17T19:07:03.5809762Z * [new branch] gh/pavithranrao/4/base -> origin/gh/pavithranrao/4/base 2022-03-17T19:07:03.5810354Z * [new branch] gh/pavithranrao/4/head -> origin/gh/pavithranrao/4/head 2022-03-17T19:07:03.5812649Z * [new branch] gh/pavithranrao/40/base -> origin/gh/pavithranrao/40/base 2022-03-17T19:07:03.5813765Z * [new branch] gh/pavithranrao/40/head -> origin/gh/pavithranrao/40/head 2022-03-17T19:07:03.5816072Z * [new branch] gh/pavithranrao/40/orig -> origin/gh/pavithranrao/40/orig 2022-03-17T19:07:03.5817716Z * [new branch] gh/pavithranrao/41/base -> origin/gh/pavithranrao/41/base 2022-03-17T19:07:03.5819007Z * [new branch] gh/pavithranrao/41/head -> origin/gh/pavithranrao/41/head 2022-03-17T19:07:03.5820774Z * [new branch] gh/pavithranrao/42/base -> origin/gh/pavithranrao/42/base 2022-03-17T19:07:03.5822794Z * [new branch] gh/pavithranrao/42/head -> origin/gh/pavithranrao/42/head 2022-03-17T19:07:03.5824011Z * [new branch] gh/pavithranrao/42/orig -> origin/gh/pavithranrao/42/orig 2022-03-17T19:07:03.5826039Z * [new branch] gh/pavithranrao/45/base -> origin/gh/pavithranrao/45/base 2022-03-17T19:07:03.5827195Z * [new branch] gh/pavithranrao/45/head -> origin/gh/pavithranrao/45/head 2022-03-17T19:07:03.5828592Z * [new branch] gh/pavithranrao/45/orig -> origin/gh/pavithranrao/45/orig 2022-03-17T19:07:03.5830652Z * [new branch] gh/pavithranrao/46/base -> origin/gh/pavithranrao/46/base 2022-03-17T19:07:03.5831862Z * [new branch] gh/pavithranrao/46/head -> origin/gh/pavithranrao/46/head 2022-03-17T19:07:03.5833644Z * [new branch] gh/pavithranrao/47/base -> origin/gh/pavithranrao/47/base 2022-03-17T19:07:03.5834712Z * [new branch] gh/pavithranrao/47/head -> origin/gh/pavithranrao/47/head 2022-03-17T19:07:03.5836135Z * [new branch] gh/pavithranrao/47/orig -> origin/gh/pavithranrao/47/orig 2022-03-17T19:07:03.5838178Z * [new branch] gh/pavithranrao/48/base -> origin/gh/pavithranrao/48/base 2022-03-17T19:07:03.5839319Z * [new branch] gh/pavithranrao/48/head -> origin/gh/pavithranrao/48/head 2022-03-17T19:07:03.5840693Z * [new branch] gh/pavithranrao/48/orig -> origin/gh/pavithranrao/48/orig 2022-03-17T19:07:03.5842715Z * [new branch] gh/pavithranrao/5/base -> origin/gh/pavithranrao/5/base 2022-03-17T19:07:03.5843782Z * [new branch] gh/pavithranrao/5/head -> origin/gh/pavithranrao/5/head 2022-03-17T19:07:03.5845756Z * [new branch] gh/pavithranrao/50/base -> origin/gh/pavithranrao/50/base 2022-03-17T19:07:03.5846944Z * [new branch] gh/pavithranrao/50/head -> origin/gh/pavithranrao/50/head 2022-03-17T19:07:03.5848722Z * [new branch] gh/pavithranrao/50/orig -> origin/gh/pavithranrao/50/orig 2022-03-17T19:07:03.5850610Z * [new branch] gh/pavithranrao/51/base -> origin/gh/pavithranrao/51/base 2022-03-17T19:07:03.5852731Z * [new branch] gh/pavithranrao/51/head -> origin/gh/pavithranrao/51/head 2022-03-17T19:07:03.5854131Z * [new branch] gh/pavithranrao/51/orig -> origin/gh/pavithranrao/51/orig 2022-03-17T19:07:03.5855829Z * [new branch] gh/pavithranrao/6/base -> origin/gh/pavithranrao/6/base 2022-03-17T19:07:03.5857028Z * [new branch] gh/pavithranrao/6/head -> origin/gh/pavithranrao/6/head 2022-03-17T19:07:03.5859341Z * [new branch] gh/pavithranrao/7/base -> origin/gh/pavithranrao/7/base 2022-03-17T19:07:03.5860730Z * [new branch] gh/pavithranrao/7/head -> origin/gh/pavithranrao/7/head 2022-03-17T19:07:03.5863394Z * [new branch] gh/pavithranrao/8/base -> origin/gh/pavithranrao/8/base 2022-03-17T19:07:03.5864432Z * [new branch] gh/pavithranrao/8/head -> origin/gh/pavithranrao/8/head 2022-03-17T19:07:03.5866649Z * [new branch] gh/pbelevich/142/base -> origin/gh/pbelevich/142/base 2022-03-17T19:07:03.5867963Z * [new branch] gh/pbelevich/142/head -> origin/gh/pbelevich/142/head 2022-03-17T19:07:03.5869306Z * [new branch] gh/pbelevich/142/orig -> origin/gh/pbelevich/142/orig 2022-03-17T19:07:03.5871208Z * [new branch] gh/pbelevich/143/base -> origin/gh/pbelevich/143/base 2022-03-17T19:07:03.5872340Z * [new branch] gh/pbelevich/143/head -> origin/gh/pbelevich/143/head 2022-03-17T19:07:03.5873685Z * [new branch] gh/pbelevich/143/orig -> origin/gh/pbelevich/143/orig 2022-03-17T19:07:03.5875305Z * [new branch] gh/pbelevich/144/base -> origin/gh/pbelevich/144/base 2022-03-17T19:07:03.5876714Z * [new branch] gh/pbelevich/144/head -> origin/gh/pbelevich/144/head 2022-03-17T19:07:03.5878020Z * [new branch] gh/pbelevich/144/orig -> origin/gh/pbelevich/144/orig 2022-03-17T19:07:03.5880382Z * [new branch] gh/pbelevich/146/base -> origin/gh/pbelevich/146/base 2022-03-17T19:07:03.5881600Z * [new branch] gh/pbelevich/146/head -> origin/gh/pbelevich/146/head 2022-03-17T19:07:03.5883044Z * [new branch] gh/pbelevich/146/orig -> origin/gh/pbelevich/146/orig 2022-03-17T19:07:03.5885087Z * [new branch] gh/pbelevich/153/base -> origin/gh/pbelevich/153/base 2022-03-17T19:07:03.5886303Z * [new branch] gh/pbelevich/153/head -> origin/gh/pbelevich/153/head 2022-03-17T19:07:03.5887680Z * [new branch] gh/pbelevich/153/orig -> origin/gh/pbelevich/153/orig 2022-03-17T19:07:03.5889662Z * [new branch] gh/pbelevich/155/base -> origin/gh/pbelevich/155/base 2022-03-17T19:07:03.5890817Z * [new branch] gh/pbelevich/155/head -> origin/gh/pbelevich/155/head 2022-03-17T19:07:03.5892217Z * [new branch] gh/pbelevich/155/orig -> origin/gh/pbelevich/155/orig 2022-03-17T19:07:03.5894206Z * [new branch] gh/pbelevich/157/base -> origin/gh/pbelevich/157/base 2022-03-17T19:07:03.5895441Z * [new branch] gh/pbelevich/157/head -> origin/gh/pbelevich/157/head 2022-03-17T19:07:03.5896793Z * [new branch] gh/pbelevich/157/orig -> origin/gh/pbelevich/157/orig 2022-03-17T19:07:03.5898817Z * [new branch] gh/pbelevich/82/head -> origin/gh/pbelevich/82/head 2022-03-17T19:07:03.5900114Z * [new branch] gh/pbelevich/82/orig -> origin/gh/pbelevich/82/orig 2022-03-17T19:07:03.5902894Z * [new branch] gh/pearu/20/base -> origin/gh/pearu/20/base 2022-03-17T19:07:03.5904092Z * [new branch] gh/pearu/20/head -> origin/gh/pearu/20/head 2022-03-17T19:07:03.5905501Z * [new branch] gh/pearu/20/orig -> origin/gh/pearu/20/orig 2022-03-17T19:07:03.5907493Z * [new branch] gh/pearu/22/base -> origin/gh/pearu/22/base 2022-03-17T19:07:03.5908576Z * [new branch] gh/pearu/22/head -> origin/gh/pearu/22/head 2022-03-17T19:07:03.5909935Z * [new branch] gh/pearu/22/orig -> origin/gh/pearu/22/orig 2022-03-17T19:07:03.5911868Z * [new branch] gh/pearu/24/base -> origin/gh/pearu/24/base 2022-03-17T19:07:03.5912825Z * [new branch] gh/pearu/24/head -> origin/gh/pearu/24/head 2022-03-17T19:07:03.5914154Z * [new branch] gh/pearu/24/orig -> origin/gh/pearu/24/orig 2022-03-17T19:07:03.5915805Z * [new branch] gh/pearu/25/base -> origin/gh/pearu/25/base 2022-03-17T19:07:03.5917105Z * [new branch] gh/pearu/25/head -> origin/gh/pearu/25/head 2022-03-17T19:07:03.5918607Z * [new branch] gh/pearu/25/orig -> origin/gh/pearu/25/orig 2022-03-17T19:07:03.5920539Z * [new branch] gh/pearu/32/base -> origin/gh/pearu/32/base 2022-03-17T19:07:03.5921667Z * [new branch] gh/pearu/32/head -> origin/gh/pearu/32/head 2022-03-17T19:07:03.5923013Z * [new branch] gh/pearu/32/orig -> origin/gh/pearu/32/orig 2022-03-17T19:07:03.5925039Z * [new branch] gh/pearu/35/base -> origin/gh/pearu/35/base 2022-03-17T19:07:03.5926222Z * [new branch] gh/pearu/35/head -> origin/gh/pearu/35/head 2022-03-17T19:07:03.5927554Z * [new branch] gh/pearu/35/orig -> origin/gh/pearu/35/orig 2022-03-17T19:07:03.5930081Z * [new branch] gh/pearu/41/base -> origin/gh/pearu/41/base 2022-03-17T19:07:03.5931227Z * [new branch] gh/pearu/41/head -> origin/gh/pearu/41/head 2022-03-17T19:07:03.5932692Z * [new branch] gh/pearu/41/orig -> origin/gh/pearu/41/orig 2022-03-17T19:07:03.5935009Z * [new branch] gh/penguinwu/10/base -> origin/gh/penguinwu/10/base 2022-03-17T19:07:03.5936274Z * [new branch] gh/penguinwu/10/head -> origin/gh/penguinwu/10/head 2022-03-17T19:07:03.5938213Z * [new branch] gh/penguinwu/10/orig -> origin/gh/penguinwu/10/orig 2022-03-17T19:07:03.5940214Z * [new branch] gh/penguinwu/11/base -> origin/gh/penguinwu/11/base 2022-03-17T19:07:03.5941302Z * [new branch] gh/penguinwu/11/head -> origin/gh/penguinwu/11/head 2022-03-17T19:07:03.5943503Z * [new branch] gh/penguinwu/11/orig -> origin/gh/penguinwu/11/orig 2022-03-17T19:07:03.5945351Z * [new branch] gh/penguinwu/7/base -> origin/gh/penguinwu/7/base 2022-03-17T19:07:03.5946514Z * [new branch] gh/penguinwu/7/head -> origin/gh/penguinwu/7/head 2022-03-17T19:07:03.5947898Z * [new branch] gh/penguinwu/7/orig -> origin/gh/penguinwu/7/orig 2022-03-17T19:07:03.5950250Z * [new branch] gh/peterbell10/100/base -> origin/gh/peterbell10/100/base 2022-03-17T19:07:03.5951463Z * [new branch] gh/peterbell10/100/orig -> origin/gh/peterbell10/100/orig 2022-03-17T19:07:03.5953451Z * [new branch] gh/peterbell10/107/orig -> origin/gh/peterbell10/107/orig 2022-03-17T19:07:03.5955073Z * [new branch] gh/peterbell10/109/base -> origin/gh/peterbell10/109/base 2022-03-17T19:07:03.5956952Z * [new branch] gh/peterbell10/109/head -> origin/gh/peterbell10/109/head 2022-03-17T19:07:03.5958038Z * [new branch] gh/peterbell10/109/orig -> origin/gh/peterbell10/109/orig 2022-03-17T19:07:03.5960108Z * [new branch] gh/peterbell10/112/base -> origin/gh/peterbell10/112/base 2022-03-17T19:07:03.5961126Z * [new branch] gh/peterbell10/112/head -> origin/gh/peterbell10/112/head 2022-03-17T19:07:03.5962538Z * [new branch] gh/peterbell10/112/orig -> origin/gh/peterbell10/112/orig 2022-03-17T19:07:03.5964495Z * [new branch] gh/peterbell10/157/base -> origin/gh/peterbell10/157/base 2022-03-17T19:07:03.5965672Z * [new branch] gh/peterbell10/157/head -> origin/gh/peterbell10/157/head 2022-03-17T19:07:03.5966979Z * [new branch] gh/peterbell10/157/orig -> origin/gh/peterbell10/157/orig 2022-03-17T19:07:03.5968852Z * [new branch] gh/peterbell10/158/base -> origin/gh/peterbell10/158/base 2022-03-17T19:07:03.5970017Z * [new branch] gh/peterbell10/158/head -> origin/gh/peterbell10/158/head 2022-03-17T19:07:03.5971701Z * [new branch] gh/peterbell10/158/orig -> origin/gh/peterbell10/158/orig 2022-03-17T19:07:03.5973500Z * [new branch] gh/peterbell10/161/base -> origin/gh/peterbell10/161/base 2022-03-17T19:07:03.5974776Z * [new branch] gh/peterbell10/161/head -> origin/gh/peterbell10/161/head 2022-03-17T19:07:03.5976263Z * [new branch] gh/peterbell10/161/orig -> origin/gh/peterbell10/161/orig 2022-03-17T19:07:03.5978295Z * [new branch] gh/peterbell10/174/base -> origin/gh/peterbell10/174/base 2022-03-17T19:07:03.5979580Z * [new branch] gh/peterbell10/174/head -> origin/gh/peterbell10/174/head 2022-03-17T19:07:03.5981091Z * [new branch] gh/peterbell10/174/orig -> origin/gh/peterbell10/174/orig 2022-03-17T19:07:03.5983520Z * [new branch] gh/peterbell10/175/base -> origin/gh/peterbell10/175/base 2022-03-17T19:07:03.5985420Z * [new branch] gh/peterbell10/175/head -> origin/gh/peterbell10/175/head 2022-03-17T19:07:03.5986609Z * [new branch] gh/peterbell10/175/orig -> origin/gh/peterbell10/175/orig 2022-03-17T19:07:03.5988682Z * [new branch] gh/peterbell10/18/base -> origin/gh/peterbell10/18/base 2022-03-17T19:07:03.5989870Z * [new branch] gh/peterbell10/18/head -> origin/gh/peterbell10/18/head 2022-03-17T19:07:03.5991282Z * [new branch] gh/peterbell10/18/orig -> origin/gh/peterbell10/18/orig 2022-03-17T19:07:03.5993895Z * [new branch] gh/peterbell10/181/base -> origin/gh/peterbell10/181/base 2022-03-17T19:07:03.5995011Z * [new branch] gh/peterbell10/181/head -> origin/gh/peterbell10/181/head 2022-03-17T19:07:03.5996336Z * [new branch] gh/peterbell10/181/orig -> origin/gh/peterbell10/181/orig 2022-03-17T19:07:03.5998652Z * [new branch] gh/peterbell10/186/base -> origin/gh/peterbell10/186/base 2022-03-17T19:07:03.5999971Z * [new branch] gh/peterbell10/186/head -> origin/gh/peterbell10/186/head 2022-03-17T19:07:03.6001450Z * [new branch] gh/peterbell10/186/orig -> origin/gh/peterbell10/186/orig 2022-03-17T19:07:03.6003477Z * [new branch] gh/peterbell10/187/base -> origin/gh/peterbell10/187/base 2022-03-17T19:07:03.6004725Z * [new branch] gh/peterbell10/187/head -> origin/gh/peterbell10/187/head 2022-03-17T19:07:03.6006032Z * [new branch] gh/peterbell10/187/orig -> origin/gh/peterbell10/187/orig 2022-03-17T19:07:03.6008698Z * [new branch] gh/peterbell10/188/base -> origin/gh/peterbell10/188/base 2022-03-17T19:07:03.6009930Z * [new branch] gh/peterbell10/188/head -> origin/gh/peterbell10/188/head 2022-03-17T19:07:03.6011305Z * [new branch] gh/peterbell10/188/orig -> origin/gh/peterbell10/188/orig 2022-03-17T19:07:03.6013592Z * [new branch] gh/peterbell10/189/base -> origin/gh/peterbell10/189/base 2022-03-17T19:07:03.6014790Z * [new branch] gh/peterbell10/189/head -> origin/gh/peterbell10/189/head 2022-03-17T19:07:03.6016279Z * [new branch] gh/peterbell10/189/orig -> origin/gh/peterbell10/189/orig 2022-03-17T19:07:03.6018700Z * [new branch] gh/peterbell10/202/base -> origin/gh/peterbell10/202/base 2022-03-17T19:07:03.6019724Z * [new branch] gh/peterbell10/202/head -> origin/gh/peterbell10/202/head 2022-03-17T19:07:03.6021118Z * [new branch] gh/peterbell10/202/orig -> origin/gh/peterbell10/202/orig 2022-03-17T19:07:03.6023754Z * [new branch] gh/peterbell10/206/base -> origin/gh/peterbell10/206/base 2022-03-17T19:07:03.6024797Z * [new branch] gh/peterbell10/206/head -> origin/gh/peterbell10/206/head 2022-03-17T19:07:03.6026160Z * [new branch] gh/peterbell10/206/orig -> origin/gh/peterbell10/206/orig 2022-03-17T19:07:03.6028278Z * [new branch] gh/peterbell10/207/base -> origin/gh/peterbell10/207/base 2022-03-17T19:07:03.6029323Z * [new branch] gh/peterbell10/207/head -> origin/gh/peterbell10/207/head 2022-03-17T19:07:03.6030733Z * [new branch] gh/peterbell10/207/orig -> origin/gh/peterbell10/207/orig 2022-03-17T19:07:03.6032935Z * [new branch] gh/peterbell10/208/base -> origin/gh/peterbell10/208/base 2022-03-17T19:07:03.6033951Z * [new branch] gh/peterbell10/208/head -> origin/gh/peterbell10/208/head 2022-03-17T19:07:03.6035221Z * [new branch] gh/peterbell10/208/orig -> origin/gh/peterbell10/208/orig 2022-03-17T19:07:03.6037469Z * [new branch] gh/peterbell10/209/base -> origin/gh/peterbell10/209/base 2022-03-17T19:07:03.6038630Z * [new branch] gh/peterbell10/209/head -> origin/gh/peterbell10/209/head 2022-03-17T19:07:03.6039876Z * [new branch] gh/peterbell10/209/orig -> origin/gh/peterbell10/209/orig 2022-03-17T19:07:03.6042013Z * [new branch] gh/peterbell10/213/base -> origin/gh/peterbell10/213/base 2022-03-17T19:07:03.6043017Z * [new branch] gh/peterbell10/213/head -> origin/gh/peterbell10/213/head 2022-03-17T19:07:03.6044349Z * [new branch] gh/peterbell10/213/orig -> origin/gh/peterbell10/213/orig 2022-03-17T19:07:03.6046505Z * [new branch] gh/peterbell10/216/base -> origin/gh/peterbell10/216/base 2022-03-17T19:07:03.6047467Z * [new branch] gh/peterbell10/216/head -> origin/gh/peterbell10/216/head 2022-03-17T19:07:03.6048813Z * [new branch] gh/peterbell10/216/orig -> origin/gh/peterbell10/216/orig 2022-03-17T19:07:03.6050954Z * [new branch] gh/peterbell10/253/base -> origin/gh/peterbell10/253/base 2022-03-17T19:07:03.6052294Z * [new branch] gh/peterbell10/253/head -> origin/gh/peterbell10/253/head 2022-03-17T19:07:03.6053691Z * [new branch] gh/peterbell10/253/orig -> origin/gh/peterbell10/253/orig 2022-03-17T19:07:03.6055733Z * [new branch] gh/peterbell10/254/base -> origin/gh/peterbell10/254/base 2022-03-17T19:07:03.6056908Z * [new branch] gh/peterbell10/254/head -> origin/gh/peterbell10/254/head 2022-03-17T19:07:03.6058866Z * [new branch] gh/peterbell10/254/orig -> origin/gh/peterbell10/254/orig 2022-03-17T19:07:03.6060196Z * [new branch] gh/peterbell10/255/base -> origin/gh/peterbell10/255/base 2022-03-17T19:07:03.6063044Z * [new branch] gh/peterbell10/255/head -> origin/gh/peterbell10/255/head 2022-03-17T19:07:03.6064093Z * [new branch] gh/peterbell10/255/orig -> origin/gh/peterbell10/255/orig 2022-03-17T19:07:03.6066270Z * [new branch] gh/peterbell10/256/base -> origin/gh/peterbell10/256/base 2022-03-17T19:07:03.6067463Z * [new branch] gh/peterbell10/256/head -> origin/gh/peterbell10/256/head 2022-03-17T19:07:03.6068865Z * [new branch] gh/peterbell10/256/orig -> origin/gh/peterbell10/256/orig 2022-03-17T19:07:03.6070725Z * [new branch] gh/peterbell10/257/base -> origin/gh/peterbell10/257/base 2022-03-17T19:07:03.6071746Z * [new branch] gh/peterbell10/257/head -> origin/gh/peterbell10/257/head 2022-03-17T19:07:03.6073182Z * [new branch] gh/peterbell10/257/orig -> origin/gh/peterbell10/257/orig 2022-03-17T19:07:03.6075264Z * [new branch] gh/peterbell10/267/base -> origin/gh/peterbell10/267/base 2022-03-17T19:07:03.6076390Z * [new branch] gh/peterbell10/267/head -> origin/gh/peterbell10/267/head 2022-03-17T19:07:03.6077791Z * [new branch] gh/peterbell10/267/orig -> origin/gh/peterbell10/267/orig 2022-03-17T19:07:03.6079966Z * [new branch] gh/peterbell10/268/base -> origin/gh/peterbell10/268/base 2022-03-17T19:07:03.6081154Z * [new branch] gh/peterbell10/268/head -> origin/gh/peterbell10/268/head 2022-03-17T19:07:03.6085761Z * [new branch] gh/peterbell10/268/orig -> origin/gh/peterbell10/268/orig 2022-03-17T19:07:03.6086068Z * [new branch] gh/peterbell10/273/base -> origin/gh/peterbell10/273/base 2022-03-17T19:07:03.6086419Z * [new branch] gh/peterbell10/273/head -> origin/gh/peterbell10/273/head 2022-03-17T19:07:03.6087232Z * [new branch] gh/peterbell10/273/orig -> origin/gh/peterbell10/273/orig 2022-03-17T19:07:03.6089848Z * [new branch] gh/peterbell10/277/base -> origin/gh/peterbell10/277/base 2022-03-17T19:07:03.6091122Z * [new branch] gh/peterbell10/277/head -> origin/gh/peterbell10/277/head 2022-03-17T19:07:03.6092819Z * [new branch] gh/peterbell10/277/orig -> origin/gh/peterbell10/277/orig 2022-03-17T19:07:03.6094743Z * [new branch] gh/peterbell10/278/base -> origin/gh/peterbell10/278/base 2022-03-17T19:07:03.6096828Z * [new branch] gh/peterbell10/278/head -> origin/gh/peterbell10/278/head 2022-03-17T19:07:03.6097297Z * [new branch] gh/peterbell10/278/orig -> origin/gh/peterbell10/278/orig 2022-03-17T19:07:03.6099430Z * [new branch] gh/peterbell10/279/base -> origin/gh/peterbell10/279/base 2022-03-17T19:07:03.6100665Z * [new branch] gh/peterbell10/279/head -> origin/gh/peterbell10/279/head 2022-03-17T19:07:03.6102612Z * [new branch] gh/peterbell10/279/orig -> origin/gh/peterbell10/279/orig 2022-03-17T19:07:03.6104986Z * [new branch] gh/peterbell10/280/base -> origin/gh/peterbell10/280/base 2022-03-17T19:07:03.6105642Z * [new branch] gh/peterbell10/280/head -> origin/gh/peterbell10/280/head 2022-03-17T19:07:03.6107282Z * [new branch] gh/peterbell10/280/orig -> origin/gh/peterbell10/280/orig 2022-03-17T19:07:03.6109693Z * [new branch] gh/peterbell10/281/base -> origin/gh/peterbell10/281/base 2022-03-17T19:07:03.6110874Z * [new branch] gh/peterbell10/281/head -> origin/gh/peterbell10/281/head 2022-03-17T19:07:03.6112209Z * [new branch] gh/peterbell10/281/orig -> origin/gh/peterbell10/281/orig 2022-03-17T19:07:03.6114275Z * [new branch] gh/peterbell10/282/base -> origin/gh/peterbell10/282/base 2022-03-17T19:07:03.6115419Z * [new branch] gh/peterbell10/282/head -> origin/gh/peterbell10/282/head 2022-03-17T19:07:03.6116956Z * [new branch] gh/peterbell10/282/orig -> origin/gh/peterbell10/282/orig 2022-03-17T19:07:03.6119190Z * [new branch] gh/peterbell10/283/base -> origin/gh/peterbell10/283/base 2022-03-17T19:07:03.6120383Z * [new branch] gh/peterbell10/283/head -> origin/gh/peterbell10/283/head 2022-03-17T19:07:03.6121665Z * [new branch] gh/peterbell10/283/orig -> origin/gh/peterbell10/283/orig 2022-03-17T19:07:03.6123813Z * [new branch] gh/peterbell10/284/base -> origin/gh/peterbell10/284/base 2022-03-17T19:07:03.6124814Z * [new branch] gh/peterbell10/284/head -> origin/gh/peterbell10/284/head 2022-03-17T19:07:03.6126092Z * [new branch] gh/peterbell10/284/orig -> origin/gh/peterbell10/284/orig 2022-03-17T19:07:03.6128250Z * [new branch] gh/peterbell10/285/base -> origin/gh/peterbell10/285/base 2022-03-17T19:07:03.6129291Z * [new branch] gh/peterbell10/285/head -> origin/gh/peterbell10/285/head 2022-03-17T19:07:03.6130478Z * [new branch] gh/peterbell10/285/orig -> origin/gh/peterbell10/285/orig 2022-03-17T19:07:03.6132510Z * [new branch] gh/peterbell10/39/base -> origin/gh/peterbell10/39/base 2022-03-17T19:07:03.6133595Z * [new branch] gh/peterbell10/39/head -> origin/gh/peterbell10/39/head 2022-03-17T19:07:03.6134959Z * [new branch] gh/peterbell10/39/orig -> origin/gh/peterbell10/39/orig 2022-03-17T19:07:03.6136909Z * [new branch] gh/peterbell10/53/base -> origin/gh/peterbell10/53/base 2022-03-17T19:07:03.6138089Z * [new branch] gh/peterbell10/53/head -> origin/gh/peterbell10/53/head 2022-03-17T19:07:03.6140014Z * [new branch] gh/peterbell10/53/orig -> origin/gh/peterbell10/53/orig 2022-03-17T19:07:03.6141606Z * [new branch] gh/peterbell10/58/base -> origin/gh/peterbell10/58/base 2022-03-17T19:07:03.6143607Z * [new branch] gh/peterbell10/58/head -> origin/gh/peterbell10/58/head 2022-03-17T19:07:03.6144693Z * [new branch] gh/peterbell10/58/orig -> origin/gh/peterbell10/58/orig 2022-03-17T19:07:03.6146763Z * [new branch] gh/peterbell10/59/base -> origin/gh/peterbell10/59/base 2022-03-17T19:07:03.6147858Z * [new branch] gh/peterbell10/59/head -> origin/gh/peterbell10/59/head 2022-03-17T19:07:03.6149261Z * [new branch] gh/peterbell10/59/orig -> origin/gh/peterbell10/59/orig 2022-03-17T19:07:03.6151327Z * [new branch] gh/peterbell10/60/base -> origin/gh/peterbell10/60/base 2022-03-17T19:07:03.6152437Z * [new branch] gh/peterbell10/60/head -> origin/gh/peterbell10/60/head 2022-03-17T19:07:03.6153800Z * [new branch] gh/peterbell10/60/orig -> origin/gh/peterbell10/60/orig 2022-03-17T19:07:03.6155938Z * [new branch] gh/peterbell10/61/base -> origin/gh/peterbell10/61/base 2022-03-17T19:07:03.6157106Z * [new branch] gh/peterbell10/61/head -> origin/gh/peterbell10/61/head 2022-03-17T19:07:03.6159265Z * [new branch] gh/peterbell10/61/orig -> origin/gh/peterbell10/61/orig 2022-03-17T19:07:03.6161259Z * [new branch] gh/peterbell10/66/base -> origin/gh/peterbell10/66/base 2022-03-17T19:07:03.6165634Z * [new branch] gh/peterbell10/66/head -> origin/gh/peterbell10/66/head 2022-03-17T19:07:03.6166258Z * [new branch] gh/peterbell10/66/orig -> origin/gh/peterbell10/66/orig 2022-03-17T19:07:03.6166540Z * [new branch] gh/pietern/10/base -> origin/gh/pietern/10/base 2022-03-17T19:07:03.6167272Z * [new branch] gh/pietern/10/head -> origin/gh/pietern/10/head 2022-03-17T19:07:03.6168948Z * [new branch] gh/pietern/10/orig -> origin/gh/pietern/10/orig 2022-03-17T19:07:03.6170677Z * [new branch] gh/pietern/23/base -> origin/gh/pietern/23/base 2022-03-17T19:07:03.6171805Z * [new branch] gh/pietern/23/orig -> origin/gh/pietern/23/orig 2022-03-17T19:07:03.6173822Z * [new branch] gh/pietern/38/base -> origin/gh/pietern/38/base 2022-03-17T19:07:03.6175094Z * [new branch] gh/pietern/38/orig -> origin/gh/pietern/38/orig 2022-03-17T19:07:03.6176961Z * [new branch] gh/pietern/4/base -> origin/gh/pietern/4/base 2022-03-17T19:07:03.6179359Z * [new branch] gh/pmeier/1/base -> origin/gh/pmeier/1/base 2022-03-17T19:07:03.6181105Z * [new branch] gh/pmeier/1/head -> origin/gh/pmeier/1/head 2022-03-17T19:07:03.6182283Z * [new branch] gh/pmeier/1/orig -> origin/gh/pmeier/1/orig 2022-03-17T19:07:03.6184198Z * [new branch] gh/pmeier/34/base -> origin/gh/pmeier/34/base 2022-03-17T19:07:03.6185367Z * [new branch] gh/pmeier/34/orig -> origin/gh/pmeier/34/orig 2022-03-17T19:07:03.6187413Z * [new branch] gh/pmeier/40/base -> origin/gh/pmeier/40/base 2022-03-17T19:07:03.6188436Z * [new branch] gh/pmeier/40/head -> origin/gh/pmeier/40/head 2022-03-17T19:07:03.6190083Z * [new branch] gh/pmeier/40/orig -> origin/gh/pmeier/40/orig 2022-03-17T19:07:03.6191890Z * [new branch] gh/pmeier/46/base -> origin/gh/pmeier/46/base 2022-03-17T19:07:03.6193289Z * [new branch] gh/pmeier/46/head -> origin/gh/pmeier/46/head 2022-03-17T19:07:03.6194903Z * [new branch] gh/pmeier/46/orig -> origin/gh/pmeier/46/orig 2022-03-17T19:07:03.6197496Z * [new branch] gh/pmeier/47/base -> origin/gh/pmeier/47/base 2022-03-17T19:07:03.6199214Z * [new branch] gh/pmeier/47/head -> origin/gh/pmeier/47/head 2022-03-17T19:07:03.6200158Z * [new branch] gh/pmeier/47/orig -> origin/gh/pmeier/47/orig 2022-03-17T19:07:03.6202198Z * [new branch] gh/pmeier/48/base -> origin/gh/pmeier/48/base 2022-03-17T19:07:03.6203370Z * [new branch] gh/pmeier/48/head -> origin/gh/pmeier/48/head 2022-03-17T19:07:03.6204937Z * [new branch] gh/pmeier/48/orig -> origin/gh/pmeier/48/orig 2022-03-17T19:07:03.6206509Z * [new branch] gh/pmeier/49/base -> origin/gh/pmeier/49/base 2022-03-17T19:07:03.6207661Z * [new branch] gh/pmeier/49/head -> origin/gh/pmeier/49/head 2022-03-17T19:07:03.6209288Z * [new branch] gh/pmeier/49/orig -> origin/gh/pmeier/49/orig 2022-03-17T19:07:03.6211068Z * [new branch] gh/pmeier/53/base -> origin/gh/pmeier/53/base 2022-03-17T19:07:03.6212217Z * [new branch] gh/pmeier/53/head -> origin/gh/pmeier/53/head 2022-03-17T19:07:03.6213912Z * [new branch] gh/pmeier/53/orig -> origin/gh/pmeier/53/orig 2022-03-17T19:07:03.6215285Z * [new branch] gh/pmeier/54/base -> origin/gh/pmeier/54/base 2022-03-17T19:07:03.6217119Z * [new branch] gh/pmeier/54/head -> origin/gh/pmeier/54/head 2022-03-17T19:07:03.6218249Z * [new branch] gh/pmeier/54/orig -> origin/gh/pmeier/54/orig 2022-03-17T19:07:03.6220230Z * [new branch] gh/pmeier/60/base -> origin/gh/pmeier/60/base 2022-03-17T19:07:03.6221642Z * [new branch] gh/pmeier/60/head -> origin/gh/pmeier/60/head 2022-03-17T19:07:03.6223338Z * [new branch] gh/pmeier/60/orig -> origin/gh/pmeier/60/orig 2022-03-17T19:07:03.6224975Z * [new branch] gh/pmeier/61/base -> origin/gh/pmeier/61/base 2022-03-17T19:07:03.6226124Z * [new branch] gh/pmeier/61/head -> origin/gh/pmeier/61/head 2022-03-17T19:07:03.6227679Z * [new branch] gh/pmeier/61/orig -> origin/gh/pmeier/61/orig 2022-03-17T19:07:03.6229428Z * [new branch] gh/pmeier/66/base -> origin/gh/pmeier/66/base 2022-03-17T19:07:03.6230527Z * [new branch] gh/pmeier/66/head -> origin/gh/pmeier/66/head 2022-03-17T19:07:03.6232309Z * [new branch] gh/pmeier/66/orig -> origin/gh/pmeier/66/orig 2022-03-17T19:07:03.6233602Z * [new branch] gh/pmeier/67/base -> origin/gh/pmeier/67/base 2022-03-17T19:07:03.6234992Z * [new branch] gh/pmeier/67/head -> origin/gh/pmeier/67/head 2022-03-17T19:07:03.6236963Z * [new branch] gh/pmeier/67/orig -> origin/gh/pmeier/67/orig 2022-03-17T19:07:03.6238395Z * [new branch] gh/pmeier/70/base -> origin/gh/pmeier/70/base 2022-03-17T19:07:03.6239731Z * [new branch] gh/pmeier/70/head -> origin/gh/pmeier/70/head 2022-03-17T19:07:03.6241115Z * [new branch] gh/pmeier/70/orig -> origin/gh/pmeier/70/orig 2022-03-17T19:07:03.6243434Z * [new branch] gh/pritamdamania87/102/base -> origin/gh/pritamdamania87/102/base 2022-03-17T19:07:03.6244567Z * [new branch] gh/pritamdamania87/102/head -> origin/gh/pritamdamania87/102/head 2022-03-17T19:07:03.6246573Z * [new branch] gh/pritamdamania87/133/base -> origin/gh/pritamdamania87/133/base 2022-03-17T19:07:03.6247761Z * [new branch] gh/pritamdamania87/133/head -> origin/gh/pritamdamania87/133/head 2022-03-17T19:07:03.6249573Z * [new branch] gh/pritamdamania87/134/base -> origin/gh/pritamdamania87/134/base 2022-03-17T19:07:03.6250622Z * [new branch] gh/pritamdamania87/134/head -> origin/gh/pritamdamania87/134/head 2022-03-17T19:07:03.6252158Z * [new branch] gh/pritamdamania87/135/base -> origin/gh/pritamdamania87/135/base 2022-03-17T19:07:03.6253377Z * [new branch] gh/pritamdamania87/135/head -> origin/gh/pritamdamania87/135/head 2022-03-17T19:07:03.6255220Z * [new branch] gh/pritamdamania87/136/base -> origin/gh/pritamdamania87/136/base 2022-03-17T19:07:03.6257027Z * [new branch] gh/pritamdamania87/136/head -> origin/gh/pritamdamania87/136/head 2022-03-17T19:07:03.6258258Z * [new branch] gh/pritamdamania87/137/base -> origin/gh/pritamdamania87/137/base 2022-03-17T19:07:03.6259893Z * [new branch] gh/pritamdamania87/137/head -> origin/gh/pritamdamania87/137/head 2022-03-17T19:07:03.6261066Z * [new branch] gh/pritamdamania87/138/base -> origin/gh/pritamdamania87/138/base 2022-03-17T19:07:03.6263768Z * [new branch] gh/pritamdamania87/138/head -> origin/gh/pritamdamania87/138/head 2022-03-17T19:07:03.6264967Z * [new branch] gh/pritamdamania87/139/base -> origin/gh/pritamdamania87/139/base 2022-03-17T19:07:03.6266579Z * [new branch] gh/pritamdamania87/139/head -> origin/gh/pritamdamania87/139/head 2022-03-17T19:07:03.6268935Z * [new branch] gh/pritamdamania87/157/base -> origin/gh/pritamdamania87/157/base 2022-03-17T19:07:03.6270145Z * [new branch] gh/pritamdamania87/157/head -> origin/gh/pritamdamania87/157/head 2022-03-17T19:07:03.6271884Z * [new branch] gh/pritamdamania87/157/orig -> origin/gh/pritamdamania87/157/orig 2022-03-17T19:07:03.6273997Z * [new branch] gh/pritamdamania87/224/base -> origin/gh/pritamdamania87/224/base 2022-03-17T19:07:03.6275160Z * [new branch] gh/pritamdamania87/224/head -> origin/gh/pritamdamania87/224/head 2022-03-17T19:07:03.6276742Z * [new branch] gh/pritamdamania87/224/orig -> origin/gh/pritamdamania87/224/orig 2022-03-17T19:07:03.6278571Z * [new branch] gh/pritamdamania87/225/base -> origin/gh/pritamdamania87/225/base 2022-03-17T19:07:03.6279780Z * [new branch] gh/pritamdamania87/225/head -> origin/gh/pritamdamania87/225/head 2022-03-17T19:07:03.6281421Z * [new branch] gh/pritamdamania87/225/orig -> origin/gh/pritamdamania87/225/orig 2022-03-17T19:07:03.6283187Z * [new branch] gh/pritamdamania87/39/base -> origin/gh/pritamdamania87/39/base 2022-03-17T19:07:03.6284417Z * [new branch] gh/pritamdamania87/39/head -> origin/gh/pritamdamania87/39/head 2022-03-17T19:07:03.6285810Z * [new branch] gh/pritamdamania87/39/orig -> origin/gh/pritamdamania87/39/orig 2022-03-17T19:07:03.6288068Z * [new branch] gh/priyaramani/5/base -> origin/gh/priyaramani/5/base 2022-03-17T19:07:03.6289238Z * [new branch] gh/priyaramani/5/head -> origin/gh/priyaramani/5/head 2022-03-17T19:07:03.6291209Z * [new branch] gh/priyaramani/6/base -> origin/gh/priyaramani/6/base 2022-03-17T19:07:03.6292362Z * [new branch] gh/priyaramani/6/head -> origin/gh/priyaramani/6/head 2022-03-17T19:07:03.6294125Z * [new branch] gh/priyaramani/7/base -> origin/gh/priyaramani/7/base 2022-03-17T19:07:03.6295133Z * [new branch] gh/priyaramani/7/head -> origin/gh/priyaramani/7/head 2022-03-17T19:07:03.6297097Z * [new branch] gh/priyaramani/9/base -> origin/gh/priyaramani/9/base 2022-03-17T19:07:03.6298249Z * [new branch] gh/priyaramani/9/head -> origin/gh/priyaramani/9/head 2022-03-17T19:07:03.6299828Z * [new branch] gh/priyaramani/9/orig -> origin/gh/priyaramani/9/orig 2022-03-17T19:07:03.6302595Z * [new branch] gh/qihan/1/base -> origin/gh/qihan/1/base 2022-03-17T19:07:03.6303835Z * [new branch] gh/qihan/1/head -> origin/gh/qihan/1/head 2022-03-17T19:07:03.6305419Z * [new branch] gh/qihan/1/orig -> origin/gh/qihan/1/orig 2022-03-17T19:07:03.6307498Z * [new branch] gh/raghuraman_k/1/base -> origin/gh/raghuraman_k/1/base 2022-03-17T19:07:03.6309655Z * [new branch] gh/raghuraman_k/1/head -> origin/gh/raghuraman_k/1/head 2022-03-17T19:07:03.6311772Z * [new branch] gh/raghuraman_k/11/base -> origin/gh/raghuraman_k/11/base 2022-03-17T19:07:03.6312959Z * [new branch] gh/raghuraman_k/11/head -> origin/gh/raghuraman_k/11/head 2022-03-17T19:07:03.6314677Z * [new branch] gh/raghuraman_k/11/orig -> origin/gh/raghuraman_k/11/orig 2022-03-17T19:07:03.6316934Z * [new branch] gh/raghuraman_k/13/base -> origin/gh/raghuraman_k/13/base 2022-03-17T19:07:03.6318134Z * [new branch] gh/raghuraman_k/13/head -> origin/gh/raghuraman_k/13/head 2022-03-17T19:07:03.6319921Z * [new branch] gh/raghuraman_k/13/orig -> origin/gh/raghuraman_k/13/orig 2022-03-17T19:07:03.6321818Z * [new branch] gh/raghuraman_k/15/base -> origin/gh/raghuraman_k/15/base 2022-03-17T19:07:03.6322791Z * [new branch] gh/raghuraman_k/15/head -> origin/gh/raghuraman_k/15/head 2022-03-17T19:07:03.6324526Z * [new branch] gh/raghuraman_k/15/orig -> origin/gh/raghuraman_k/15/orig 2022-03-17T19:07:03.6326299Z * [new branch] gh/raghuraman_k/17/base -> origin/gh/raghuraman_k/17/base 2022-03-17T19:07:03.6328308Z * [new branch] gh/raghuraman_k/17/head -> origin/gh/raghuraman_k/17/head 2022-03-17T19:07:03.6329970Z * [new branch] gh/raghuraman_k/17/orig -> origin/gh/raghuraman_k/17/orig 2022-03-17T19:07:03.6331962Z * [new branch] gh/raghuraman_k/19/base -> origin/gh/raghuraman_k/19/base 2022-03-17T19:07:03.6333020Z * [new branch] gh/raghuraman_k/19/head -> origin/gh/raghuraman_k/19/head 2022-03-17T19:07:03.6334721Z * [new branch] gh/raghuraman_k/19/orig -> origin/gh/raghuraman_k/19/orig 2022-03-17T19:07:03.6335965Z * [new branch] gh/raghuraman_k/2/base -> origin/gh/raghuraman_k/2/base 2022-03-17T19:07:03.6337354Z * [new branch] gh/raghuraman_k/2/head -> origin/gh/raghuraman_k/2/head 2022-03-17T19:07:03.6339221Z * [new branch] gh/raghuraman_k/20/base -> origin/gh/raghuraman_k/20/base 2022-03-17T19:07:03.6340599Z * [new branch] gh/raghuraman_k/20/head -> origin/gh/raghuraman_k/20/head 2022-03-17T19:07:03.6342561Z * [new branch] gh/raghuraman_k/20/orig -> origin/gh/raghuraman_k/20/orig 2022-03-17T19:07:03.6344448Z * [new branch] gh/raghuraman_k/21/base -> origin/gh/raghuraman_k/21/base 2022-03-17T19:07:03.6345587Z * [new branch] gh/raghuraman_k/21/head -> origin/gh/raghuraman_k/21/head 2022-03-17T19:07:03.6347201Z * [new branch] gh/raghuraman_k/21/orig -> origin/gh/raghuraman_k/21/orig 2022-03-17T19:07:03.6348981Z * [new branch] gh/raghuraman_k/4/base -> origin/gh/raghuraman_k/4/base 2022-03-17T19:07:03.6350652Z * [new branch] gh/raghuraman_k/4/head -> origin/gh/raghuraman_k/4/head 2022-03-17T19:07:03.6351866Z * [new branch] gh/raghuraman_k/4/orig -> origin/gh/raghuraman_k/4/orig 2022-03-17T19:07:03.6353932Z * [new branch] gh/raghuraman_k/5/base -> origin/gh/raghuraman_k/5/base 2022-03-17T19:07:03.6355549Z * [new branch] gh/raghuraman_k/5/head -> origin/gh/raghuraman_k/5/head 2022-03-17T19:07:03.6356689Z * [new branch] gh/raghuraman_k/5/orig -> origin/gh/raghuraman_k/5/orig 2022-03-17T19:07:03.6358702Z * [new branch] gh/raghuraman_k/6/base -> origin/gh/raghuraman_k/6/base 2022-03-17T19:07:03.6359821Z * [new branch] gh/raghuraman_k/6/head -> origin/gh/raghuraman_k/6/head 2022-03-17T19:07:03.6361160Z * [new branch] gh/raghuraman_k/6/orig -> origin/gh/raghuraman_k/6/orig 2022-03-17T19:07:03.6362994Z * [new branch] gh/raghuraman_k/7/base -> origin/gh/raghuraman_k/7/base 2022-03-17T19:07:03.6363989Z * [new branch] gh/raghuraman_k/7/head -> origin/gh/raghuraman_k/7/head 2022-03-17T19:07:03.6365221Z * [new branch] gh/raghuraman_k/7/orig -> origin/gh/raghuraman_k/7/orig 2022-03-17T19:07:03.6367539Z * [new branch] gh/raghuraman_k/9/base -> origin/gh/raghuraman_k/9/base 2022-03-17T19:07:03.6368594Z * [new branch] gh/raghuraman_k/9/head -> origin/gh/raghuraman_k/9/head 2022-03-17T19:07:03.6370465Z * [new branch] gh/raghuraman_k/9/orig -> origin/gh/raghuraman_k/9/orig 2022-03-17T19:07:03.6372474Z * [new branch] gh/raghuramank100/10/base -> origin/gh/raghuramank100/10/base 2022-03-17T19:07:03.6373687Z * [new branch] gh/raghuramank100/10/head -> origin/gh/raghuramank100/10/head 2022-03-17T19:07:03.6375680Z * [new branch] gh/raghuramank100/10/orig -> origin/gh/raghuramank100/10/orig 2022-03-17T19:07:03.6376677Z * [new branch] gh/raghuramank100/11/base -> origin/gh/raghuramank100/11/base 2022-03-17T19:07:03.6378070Z * [new branch] gh/raghuramank100/11/head -> origin/gh/raghuramank100/11/head 2022-03-17T19:07:03.6379880Z * [new branch] gh/raghuramank100/11/orig -> origin/gh/raghuramank100/11/orig 2022-03-17T19:07:03.6381175Z * [new branch] gh/raghuramank100/12/base -> origin/gh/raghuramank100/12/base 2022-03-17T19:07:03.6383183Z * [new branch] gh/raghuramank100/12/head -> origin/gh/raghuramank100/12/head 2022-03-17T19:07:03.6384159Z * [new branch] gh/raghuramank100/12/orig -> origin/gh/raghuramank100/12/orig 2022-03-17T19:07:03.6385692Z * [new branch] gh/raghuramank100/16/base -> origin/gh/raghuramank100/16/base 2022-03-17T19:07:03.6387000Z * [new branch] gh/raghuramank100/16/head -> origin/gh/raghuramank100/16/head 2022-03-17T19:07:03.6388994Z * [new branch] gh/raghuramank100/19/base -> origin/gh/raghuramank100/19/base 2022-03-17T19:07:03.6390124Z * [new branch] gh/raghuramank100/19/head -> origin/gh/raghuramank100/19/head 2022-03-17T19:07:03.6391557Z * [new branch] gh/raghuramank100/19/orig -> origin/gh/raghuramank100/19/orig 2022-03-17T19:07:03.6393625Z * [new branch] gh/raghuramank100/22/base -> origin/gh/raghuramank100/22/base 2022-03-17T19:07:03.6394829Z * [new branch] gh/raghuramank100/22/head -> origin/gh/raghuramank100/22/head 2022-03-17T19:07:03.6396554Z * [new branch] gh/raghuramank100/22/orig -> origin/gh/raghuramank100/22/orig 2022-03-17T19:07:03.6397839Z * [new branch] gh/raghuramank100/26/base -> origin/gh/raghuramank100/26/base 2022-03-17T19:07:03.6399461Z * [new branch] gh/raghuramank100/26/head -> origin/gh/raghuramank100/26/head 2022-03-17T19:07:03.6400628Z * [new branch] gh/raghuramank100/26/orig -> origin/gh/raghuramank100/26/orig 2022-03-17T19:07:03.6402601Z * [new branch] gh/raghuramank100/28/base -> origin/gh/raghuramank100/28/base 2022-03-17T19:07:03.6403879Z * [new branch] gh/raghuramank100/28/head -> origin/gh/raghuramank100/28/head 2022-03-17T19:07:03.6405669Z * [new branch] gh/raghuramank100/28/orig -> origin/gh/raghuramank100/28/orig 2022-03-17T19:07:03.6407385Z * [new branch] gh/raghuramank100/30/base -> origin/gh/raghuramank100/30/base 2022-03-17T19:07:03.6408599Z * [new branch] gh/raghuramank100/30/head -> origin/gh/raghuramank100/30/head 2022-03-17T19:07:03.6410242Z * [new branch] gh/raghuramank100/30/orig -> origin/gh/raghuramank100/30/orig 2022-03-17T19:07:03.6412093Z * [new branch] gh/raghuramank100/37/base -> origin/gh/raghuramank100/37/base 2022-03-17T19:07:03.6413279Z * [new branch] gh/raghuramank100/37/head -> origin/gh/raghuramank100/37/head 2022-03-17T19:07:03.6415009Z * [new branch] gh/raghuramank100/37/orig -> origin/gh/raghuramank100/37/orig 2022-03-17T19:07:03.6416913Z * [new branch] gh/raghuramank100/39/base -> origin/gh/raghuramank100/39/base 2022-03-17T19:07:03.6418088Z * [new branch] gh/raghuramank100/39/head -> origin/gh/raghuramank100/39/head 2022-03-17T19:07:03.6419855Z * [new branch] gh/raghuramank100/39/orig -> origin/gh/raghuramank100/39/orig 2022-03-17T19:07:03.6421925Z * [new branch] gh/raghuramank100/4/base -> origin/gh/raghuramank100/4/base 2022-03-17T19:07:03.6423180Z * [new branch] gh/raghuramank100/4/head -> origin/gh/raghuramank100/4/head 2022-03-17T19:07:03.6424894Z * [new branch] gh/raghuramank100/4/orig -> origin/gh/raghuramank100/4/orig 2022-03-17T19:07:03.6426637Z * [new branch] gh/raghuramank100/40/base -> origin/gh/raghuramank100/40/base 2022-03-17T19:07:03.6428341Z * [new branch] gh/raghuramank100/40/head -> origin/gh/raghuramank100/40/head 2022-03-17T19:07:03.6429398Z * [new branch] gh/raghuramank100/40/orig -> origin/gh/raghuramank100/40/orig 2022-03-17T19:07:03.6431201Z * [new branch] gh/raghuramank100/41/base -> origin/gh/raghuramank100/41/base 2022-03-17T19:07:03.6432411Z * [new branch] gh/raghuramank100/41/head -> origin/gh/raghuramank100/41/head 2022-03-17T19:07:03.6434116Z * [new branch] gh/raghuramank100/41/orig -> origin/gh/raghuramank100/41/orig 2022-03-17T19:07:03.6435389Z * [new branch] gh/raghuramank100/42/base -> origin/gh/raghuramank100/42/base 2022-03-17T19:07:03.6437488Z * [new branch] gh/raghuramank100/42/head -> origin/gh/raghuramank100/42/head 2022-03-17T19:07:03.6438466Z * [new branch] gh/raghuramank100/42/orig -> origin/gh/raghuramank100/42/orig 2022-03-17T19:07:03.6440565Z * [new branch] gh/raghuramank100/43/base -> origin/gh/raghuramank100/43/base 2022-03-17T19:07:03.6441733Z * [new branch] gh/raghuramank100/43/head -> origin/gh/raghuramank100/43/head 2022-03-17T19:07:03.6443378Z * [new branch] gh/raghuramank100/43/orig -> origin/gh/raghuramank100/43/orig 2022-03-17T19:07:03.6445124Z * [new branch] gh/raghuramank100/46/base -> origin/gh/raghuramank100/46/base 2022-03-17T19:07:03.6446289Z * [new branch] gh/raghuramank100/46/head -> origin/gh/raghuramank100/46/head 2022-03-17T19:07:03.6448166Z * [new branch] gh/raghuramank100/46/orig -> origin/gh/raghuramank100/46/orig 2022-03-17T19:07:03.6450016Z * [new branch] gh/raghuramank100/47/base -> origin/gh/raghuramank100/47/base 2022-03-17T19:07:03.6451645Z * [new branch] gh/raghuramank100/47/head -> origin/gh/raghuramank100/47/head 2022-03-17T19:07:03.6453038Z * [new branch] gh/raghuramank100/47/orig -> origin/gh/raghuramank100/47/orig 2022-03-17T19:07:03.6455077Z * [new branch] gh/raghuramank100/48/base -> origin/gh/raghuramank100/48/base 2022-03-17T19:07:03.6456189Z * [new branch] gh/raghuramank100/48/head -> origin/gh/raghuramank100/48/head 2022-03-17T19:07:03.6457884Z * [new branch] gh/raghuramank100/48/orig -> origin/gh/raghuramank100/48/orig 2022-03-17T19:07:03.6459629Z * [new branch] gh/raghuramank100/49/base -> origin/gh/raghuramank100/49/base 2022-03-17T19:07:03.6460894Z * [new branch] gh/raghuramank100/49/head -> origin/gh/raghuramank100/49/head 2022-03-17T19:07:03.6463778Z * [new branch] gh/raghuramank100/49/orig -> origin/gh/raghuramank100/49/orig 2022-03-17T19:07:03.6465018Z * [new branch] gh/raghuramank100/5/base -> origin/gh/raghuramank100/5/base 2022-03-17T19:07:03.6466671Z * [new branch] gh/raghuramank100/5/head -> origin/gh/raghuramank100/5/head 2022-03-17T19:07:03.6467892Z * [new branch] gh/raghuramank100/5/orig -> origin/gh/raghuramank100/5/orig 2022-03-17T19:07:03.6470163Z * [new branch] gh/raghuramank100/50/base -> origin/gh/raghuramank100/50/base 2022-03-17T19:07:03.6471382Z * [new branch] gh/raghuramank100/50/head -> origin/gh/raghuramank100/50/head 2022-03-17T19:07:03.6473193Z * [new branch] gh/raghuramank100/50/orig -> origin/gh/raghuramank100/50/orig 2022-03-17T19:07:03.6474425Z * [new branch] gh/raghuramank100/6/base -> origin/gh/raghuramank100/6/base 2022-03-17T19:07:03.6476068Z * [new branch] gh/raghuramank100/6/head -> origin/gh/raghuramank100/6/head 2022-03-17T19:07:03.6476915Z * [new branch] gh/raghuramank100/6/orig -> origin/gh/raghuramank100/6/orig 2022-03-17T19:07:03.6479026Z * [new branch] gh/raghuramank100/7/base -> origin/gh/raghuramank100/7/base 2022-03-17T19:07:03.6480603Z * [new branch] gh/raghuramank100/7/head -> origin/gh/raghuramank100/7/head 2022-03-17T19:07:03.6481782Z * [new branch] gh/raghuramank100/7/orig -> origin/gh/raghuramank100/7/orig 2022-03-17T19:07:03.6483773Z * [new branch] gh/raghuramank100/8/base -> origin/gh/raghuramank100/8/base 2022-03-17T19:07:03.6484979Z * [new branch] gh/raghuramank100/8/head -> origin/gh/raghuramank100/8/head 2022-03-17T19:07:03.6486701Z * [new branch] gh/raghuramank100/8/orig -> origin/gh/raghuramank100/8/orig 2022-03-17T19:07:03.6489209Z * [new branch] gh/robieta/11/base -> origin/gh/robieta/11/base 2022-03-17T19:07:03.6490634Z * [new branch] gh/robieta/11/head -> origin/gh/robieta/11/head 2022-03-17T19:07:03.6492282Z * [new branch] gh/robieta/12/base -> origin/gh/robieta/12/base 2022-03-17T19:07:03.6494140Z * [new branch] gh/robieta/13/base -> origin/gh/robieta/13/base 2022-03-17T19:07:03.6495960Z * [new branch] gh/robieta/22/base -> origin/gh/robieta/22/base 2022-03-17T19:07:03.6497334Z * [new branch] gh/robieta/22/head -> origin/gh/robieta/22/head 2022-03-17T19:07:03.6498701Z * [new branch] gh/robieta/22/orig -> origin/gh/robieta/22/orig 2022-03-17T19:07:03.6500915Z * [new branch] gh/robieta/24/base -> origin/gh/robieta/24/base 2022-03-17T19:07:03.6502738Z * [new branch] gh/robieta/24/head -> origin/gh/robieta/24/head 2022-03-17T19:07:03.6504006Z * [new branch] gh/robieta/24/orig -> origin/gh/robieta/24/orig 2022-03-17T19:07:03.6506565Z * [new branch] gh/robieta/25/base -> origin/gh/robieta/25/base 2022-03-17T19:07:03.6508273Z * [new branch] gh/robieta/25/head -> origin/gh/robieta/25/head 2022-03-17T19:07:03.6509455Z * [new branch] gh/robieta/25/orig -> origin/gh/robieta/25/orig 2022-03-17T19:07:03.6511593Z * [new branch] gh/robieta/26/base -> origin/gh/robieta/26/base 2022-03-17T19:07:03.6513122Z * [new branch] gh/robieta/26/head -> origin/gh/robieta/26/head 2022-03-17T19:07:03.6514277Z * [new branch] gh/robieta/26/orig -> origin/gh/robieta/26/orig 2022-03-17T19:07:03.6516608Z * [new branch] gh/robieta/30/base -> origin/gh/robieta/30/base 2022-03-17T19:07:03.6518205Z * [new branch] gh/robieta/30/head -> origin/gh/robieta/30/head 2022-03-17T19:07:03.6519254Z * [new branch] gh/robieta/30/orig -> origin/gh/robieta/30/orig 2022-03-17T19:07:03.6521478Z * [new branch] gh/robieta/31/base -> origin/gh/robieta/31/base 2022-03-17T19:07:03.6522665Z * [new branch] gh/robieta/31/head -> origin/gh/robieta/31/head 2022-03-17T19:07:03.6524324Z * [new branch] gh/robieta/31/orig -> origin/gh/robieta/31/orig 2022-03-17T19:07:03.6526267Z * [new branch] gh/robieta/32/base -> origin/gh/robieta/32/base 2022-03-17T19:07:03.6527853Z * [new branch] gh/robieta/32/head -> origin/gh/robieta/32/head 2022-03-17T19:07:03.6529639Z * [new branch] gh/robieta/32/orig -> origin/gh/robieta/32/orig 2022-03-17T19:07:03.6531285Z * [new branch] gh/robieta/33/base -> origin/gh/robieta/33/base 2022-03-17T19:07:03.6532582Z * [new branch] gh/robieta/33/head -> origin/gh/robieta/33/head 2022-03-17T19:07:03.6534233Z * [new branch] gh/robieta/33/orig -> origin/gh/robieta/33/orig 2022-03-17T19:07:03.6536670Z * [new branch] gh/robieta/34/base -> origin/gh/robieta/34/base 2022-03-17T19:07:03.6537784Z * [new branch] gh/robieta/34/head -> origin/gh/robieta/34/head 2022-03-17T19:07:03.6539156Z * [new branch] gh/robieta/34/orig -> origin/gh/robieta/34/orig 2022-03-17T19:07:03.6541415Z * [new branch] gh/robieta/35/base -> origin/gh/robieta/35/base 2022-03-17T19:07:03.6543362Z * [new branch] gh/robieta/35/head -> origin/gh/robieta/35/head 2022-03-17T19:07:03.6544543Z * [new branch] gh/robieta/35/orig -> origin/gh/robieta/35/orig 2022-03-17T19:07:03.6547376Z * [new branch] gh/robieta/36/base -> origin/gh/robieta/36/base 2022-03-17T19:07:03.6548556Z * [new branch] gh/robieta/36/head -> origin/gh/robieta/36/head 2022-03-17T19:07:03.6549967Z * [new branch] gh/robieta/36/orig -> origin/gh/robieta/36/orig 2022-03-17T19:07:03.6552208Z * [new branch] gh/robieta/37/base -> origin/gh/robieta/37/base 2022-03-17T19:07:03.6553236Z * [new branch] gh/robieta/37/head -> origin/gh/robieta/37/head 2022-03-17T19:07:03.6554604Z * [new branch] gh/robieta/37/orig -> origin/gh/robieta/37/orig 2022-03-17T19:07:03.6556043Z * [new branch] gh/robieta/benchmark_utils -> origin/gh/robieta/benchmark_utils 2022-03-17T19:07:03.6557402Z * [new branch] gh/robieta/schedule_functor -> origin/gh/robieta/schedule_functor 2022-03-17T19:07:03.6559755Z * [new branch] gh/rohan-varma/1/base -> origin/gh/rohan-varma/1/base 2022-03-17T19:07:03.6560818Z * [new branch] gh/rohan-varma/1/orig -> origin/gh/rohan-varma/1/orig 2022-03-17T19:07:03.6563588Z * [new branch] gh/rohan-varma/160/base -> origin/gh/rohan-varma/160/base 2022-03-17T19:07:03.6564765Z * [new branch] gh/rohan-varma/160/head -> origin/gh/rohan-varma/160/head 2022-03-17T19:07:03.6566675Z * [new branch] gh/rohan-varma/191/base -> origin/gh/rohan-varma/191/base 2022-03-17T19:07:03.6568070Z * [new branch] gh/rohan-varma/191/head -> origin/gh/rohan-varma/191/head 2022-03-17T19:07:03.6569319Z * [new branch] gh/rohan-varma/191/orig -> origin/gh/rohan-varma/191/orig 2022-03-17T19:07:03.6571402Z * [new branch] gh/rohan-varma/256/base -> origin/gh/rohan-varma/256/base 2022-03-17T19:07:03.6572824Z * [new branch] gh/rohan-varma/256/head -> origin/gh/rohan-varma/256/head 2022-03-17T19:07:03.6574824Z * [new branch] gh/rohan-varma/284/base -> origin/gh/rohan-varma/284/base 2022-03-17T19:07:03.6576822Z * [new branch] gh/rohan-varma/284/head -> origin/gh/rohan-varma/284/head 2022-03-17T19:07:03.6578208Z * [new branch] gh/rohan-varma/284/orig -> origin/gh/rohan-varma/284/orig 2022-03-17T19:07:03.6580180Z * [new branch] gh/rohan-varma/34/base -> origin/gh/rohan-varma/34/base 2022-03-17T19:07:03.6581982Z * [new branch] gh/rohan-varma/34/head -> origin/gh/rohan-varma/34/head 2022-03-17T19:07:03.6583121Z * [new branch] gh/rohan-varma/34/orig -> origin/gh/rohan-varma/34/orig 2022-03-17T19:07:03.6585268Z * [new branch] gh/rohan-varma/378/base -> origin/gh/rohan-varma/378/base 2022-03-17T19:07:03.6586439Z * [new branch] gh/rohan-varma/378/head -> origin/gh/rohan-varma/378/head 2022-03-17T19:07:03.6588235Z * [new branch] gh/rohan-varma/378/orig -> origin/gh/rohan-varma/378/orig 2022-03-17T19:07:03.6589608Z * [new branch] gh/rohan-varma/379/base -> origin/gh/rohan-varma/379/base 2022-03-17T19:07:03.6591881Z * [new branch] gh/rohan-varma/379/head -> origin/gh/rohan-varma/379/head 2022-03-17T19:07:03.6593270Z * [new branch] gh/rohan-varma/379/orig -> origin/gh/rohan-varma/379/orig 2022-03-17T19:07:03.6595277Z * [new branch] gh/rohan-varma/411/base -> origin/gh/rohan-varma/411/base 2022-03-17T19:07:03.6596422Z * [new branch] gh/rohan-varma/411/head -> origin/gh/rohan-varma/411/head 2022-03-17T19:07:03.6598140Z * [new branch] gh/rohan-varma/411/orig -> origin/gh/rohan-varma/411/orig 2022-03-17T19:07:03.6600085Z * [new branch] gh/rohan-varma/438/base -> origin/gh/rohan-varma/438/base 2022-03-17T19:07:03.6601735Z * [new branch] gh/rohan-varma/438/head -> origin/gh/rohan-varma/438/head 2022-03-17T19:07:03.6603361Z * [new branch] gh/rohan-varma/438/orig -> origin/gh/rohan-varma/438/orig 2022-03-17T19:07:03.6605148Z * [new branch] gh/rohan-varma/466/base -> origin/gh/rohan-varma/466/base 2022-03-17T19:07:03.6606416Z * [new branch] gh/rohan-varma/466/head -> origin/gh/rohan-varma/466/head 2022-03-17T19:07:03.6608117Z * [new branch] gh/rohan-varma/466/orig -> origin/gh/rohan-varma/466/orig 2022-03-17T19:07:03.6609442Z * [new branch] gh/rohan-varma/472/base -> origin/gh/rohan-varma/472/base 2022-03-17T19:07:03.6611641Z * [new branch] gh/rohan-varma/472/head -> origin/gh/rohan-varma/472/head 2022-03-17T19:07:03.6613501Z * [new branch] gh/rohan-varma/472/orig -> origin/gh/rohan-varma/472/orig 2022-03-17T19:07:03.6615612Z * [new branch] gh/rohan-varma/491/base -> origin/gh/rohan-varma/491/base 2022-03-17T19:07:03.6616889Z * [new branch] gh/rohan-varma/491/head -> origin/gh/rohan-varma/491/head 2022-03-17T19:07:03.6618534Z * [new branch] gh/rohan-varma/491/orig -> origin/gh/rohan-varma/491/orig 2022-03-17T19:07:03.6620356Z * [new branch] gh/rohan-varma/506/base -> origin/gh/rohan-varma/506/base 2022-03-17T19:07:03.6622856Z * [new branch] gh/rohan-varma/506/head -> origin/gh/rohan-varma/506/head 2022-03-17T19:07:03.6624674Z * [new branch] gh/rohan-varma/506/orig -> origin/gh/rohan-varma/506/orig 2022-03-17T19:07:03.6625712Z * [new branch] gh/rohan-varma/516/base -> origin/gh/rohan-varma/516/base 2022-03-17T19:07:03.6627329Z * [new branch] gh/rohan-varma/516/head -> origin/gh/rohan-varma/516/head 2022-03-17T19:07:03.6628497Z * [new branch] gh/rohan-varma/516/orig -> origin/gh/rohan-varma/516/orig 2022-03-17T19:07:03.6630471Z * [new branch] gh/rohan-varma/518/base -> origin/gh/rohan-varma/518/base 2022-03-17T19:07:03.6631657Z * [new branch] gh/rohan-varma/518/head -> origin/gh/rohan-varma/518/head 2022-03-17T19:07:03.6633249Z * [new branch] gh/rohan-varma/518/orig -> origin/gh/rohan-varma/518/orig 2022-03-17T19:07:03.6635232Z * [new branch] gh/rohan-varma/519/base -> origin/gh/rohan-varma/519/base 2022-03-17T19:07:03.6636288Z * [new branch] gh/rohan-varma/519/head -> origin/gh/rohan-varma/519/head 2022-03-17T19:07:03.6638048Z * [new branch] gh/rohan-varma/519/orig -> origin/gh/rohan-varma/519/orig 2022-03-17T19:07:03.6639761Z * [new branch] gh/rohan-varma/520/base -> origin/gh/rohan-varma/520/base 2022-03-17T19:07:03.6640952Z * [new branch] gh/rohan-varma/520/head -> origin/gh/rohan-varma/520/head 2022-03-17T19:07:03.6642604Z * [new branch] gh/rohan-varma/520/orig -> origin/gh/rohan-varma/520/orig 2022-03-17T19:07:03.6644283Z * [new branch] gh/rohan-varma/521/base -> origin/gh/rohan-varma/521/base 2022-03-17T19:07:03.6645464Z * [new branch] gh/rohan-varma/521/head -> origin/gh/rohan-varma/521/head 2022-03-17T19:07:03.6647077Z * [new branch] gh/rohan-varma/521/orig -> origin/gh/rohan-varma/521/orig 2022-03-17T19:07:03.6648924Z * [new branch] gh/rohan-varma/522/base -> origin/gh/rohan-varma/522/base 2022-03-17T19:07:03.6650397Z * [new branch] gh/rohan-varma/522/head -> origin/gh/rohan-varma/522/head 2022-03-17T19:07:03.6651614Z * [new branch] gh/rohan-varma/522/orig -> origin/gh/rohan-varma/522/orig 2022-03-17T19:07:03.6654157Z * [new branch] gh/rohan-varma/523/base -> origin/gh/rohan-varma/523/base 2022-03-17T19:07:03.6655301Z * [new branch] gh/rohan-varma/523/head -> origin/gh/rohan-varma/523/head 2022-03-17T19:07:03.6656883Z * [new branch] gh/rohan-varma/523/orig -> origin/gh/rohan-varma/523/orig 2022-03-17T19:07:03.6658263Z * [new branch] gh/rohan-varma/524/base -> origin/gh/rohan-varma/524/base 2022-03-17T19:07:03.6659647Z * [new branch] gh/rohan-varma/524/head -> origin/gh/rohan-varma/524/head 2022-03-17T19:07:03.6661187Z * [new branch] gh/rohan-varma/524/orig -> origin/gh/rohan-varma/524/orig 2022-03-17T19:07:03.6664087Z * [new branch] gh/rohan-varma/9/base -> origin/gh/rohan-varma/9/base 2022-03-17T19:07:03.6665242Z * [new branch] gh/rohan-varma/9/head -> origin/gh/rohan-varma/9/head 2022-03-17T19:07:03.6666762Z * [new branch] gh/rohan-varma/9/orig -> origin/gh/rohan-varma/9/orig 2022-03-17T19:07:03.6668856Z * [new branch] gh/rsemenov/2/base -> origin/gh/rsemenov/2/base 2022-03-17T19:07:03.6669998Z * [new branch] gh/rsemenov/2/head -> origin/gh/rsemenov/2/head 2022-03-17T19:07:03.6671645Z * [new branch] gh/rsemenov/2/orig -> origin/gh/rsemenov/2/orig 2022-03-17T19:07:03.6674030Z * [new branch] gh/sahancpal@gmail.com/1/base -> origin/gh/sahancpal@gmail.com/1/base 2022-03-17T19:07:03.6675190Z * [new branch] gh/sahancpal@gmail.com/1/orig -> origin/gh/sahancpal@gmail.com/1/orig 2022-03-17T19:07:03.6677257Z * [new branch] gh/sahancpal@gmail.com/2/base -> origin/gh/sahancpal@gmail.com/2/base 2022-03-17T19:07:03.6678437Z * [new branch] gh/sahancpal@gmail.com/2/orig -> origin/gh/sahancpal@gmail.com/2/orig 2022-03-17T19:07:03.6680418Z * [new branch] gh/sahancpal@gmail.com/3/base -> origin/gh/sahancpal@gmail.com/3/base 2022-03-17T19:07:03.6681602Z * [new branch] gh/sahancpal@gmail.com/3/orig -> origin/gh/sahancpal@gmail.com/3/orig 2022-03-17T19:07:03.6683797Z * [new branch] gh/salexspb/1/base -> origin/gh/salexspb/1/base 2022-03-17T19:07:03.6685386Z * [new branch] gh/salexspb/1/head -> origin/gh/salexspb/1/head 2022-03-17T19:07:03.6687136Z * [new branch] gh/salexspb/2/base -> origin/gh/salexspb/2/base 2022-03-17T19:07:03.6688128Z * [new branch] gh/salexspb/2/head -> origin/gh/salexspb/2/head 2022-03-17T19:07:03.6690751Z * [new branch] gh/salilsdesai/14/base -> origin/gh/salilsdesai/14/base 2022-03-17T19:07:03.6691884Z * [new branch] gh/salilsdesai/14/head -> origin/gh/salilsdesai/14/head 2022-03-17T19:07:03.6693858Z * [new branch] gh/salilsdesai/14/orig -> origin/gh/salilsdesai/14/orig 2022-03-17T19:07:03.6695241Z * [new branch] gh/salilsdesai/15/base -> origin/gh/salilsdesai/15/base 2022-03-17T19:07:03.6696615Z * [new branch] gh/salilsdesai/15/head -> origin/gh/salilsdesai/15/head 2022-03-17T19:07:03.6698005Z * [new branch] gh/salilsdesai/15/orig -> origin/gh/salilsdesai/15/orig 2022-03-17T19:07:03.6700106Z * [new branch] gh/salilsdesai/16/base -> origin/gh/salilsdesai/16/base 2022-03-17T19:07:03.6701251Z * [new branch] gh/salilsdesai/16/head -> origin/gh/salilsdesai/16/head 2022-03-17T19:07:03.6703394Z * [new branch] gh/salilsdesai/16/orig -> origin/gh/salilsdesai/16/orig 2022-03-17T19:07:03.6704716Z * [new branch] gh/salilsdesai/17/base -> origin/gh/salilsdesai/17/base 2022-03-17T19:07:03.6706139Z * [new branch] gh/salilsdesai/17/head -> origin/gh/salilsdesai/17/head 2022-03-17T19:07:03.6708520Z * [new branch] gh/salilsdesai/17/orig -> origin/gh/salilsdesai/17/orig 2022-03-17T19:07:03.6709932Z * [new branch] gh/salilsdesai/5/base -> origin/gh/salilsdesai/5/base 2022-03-17T19:07:03.6711513Z * [new branch] gh/salilsdesai/5/head -> origin/gh/salilsdesai/5/head 2022-03-17T19:07:03.6712903Z * [new branch] gh/salilsdesai/5/orig -> origin/gh/salilsdesai/5/orig 2022-03-17T19:07:03.6714603Z * [new branch] gh/salilsdesai/6/base -> origin/gh/salilsdesai/6/base 2022-03-17T19:07:03.6715846Z * [new branch] gh/salilsdesai/6/head -> origin/gh/salilsdesai/6/head 2022-03-17T19:07:03.6717335Z * [new branch] gh/salilsdesai/6/orig -> origin/gh/salilsdesai/6/orig 2022-03-17T19:07:03.6719679Z * [new branch] gh/samdow/2/base -> origin/gh/samdow/2/base 2022-03-17T19:07:03.6720691Z * [new branch] gh/samdow/2/head -> origin/gh/samdow/2/head 2022-03-17T19:07:03.6722217Z * [new branch] gh/samdow/2/orig -> origin/gh/samdow/2/orig 2022-03-17T19:07:03.6724021Z * [new branch] gh/samdow/4/base -> origin/gh/samdow/4/base 2022-03-17T19:07:03.6725066Z * [new branch] gh/samdow/4/head -> origin/gh/samdow/4/head 2022-03-17T19:07:03.6726737Z * [new branch] gh/samdow/4/orig -> origin/gh/samdow/4/orig 2022-03-17T19:07:03.6728644Z * [new branch] gh/samdow/5/base -> origin/gh/samdow/5/base 2022-03-17T19:07:03.6729542Z * [new branch] gh/samdow/5/head -> origin/gh/samdow/5/head 2022-03-17T19:07:03.6731195Z * [new branch] gh/samdow/5/orig -> origin/gh/samdow/5/orig 2022-03-17T19:07:03.6732931Z * [new branch] gh/samdow/6/base -> origin/gh/samdow/6/base 2022-03-17T19:07:03.6733864Z * [new branch] gh/samdow/6/head -> origin/gh/samdow/6/head 2022-03-17T19:07:03.6735553Z * [new branch] gh/samdow/6/orig -> origin/gh/samdow/6/orig 2022-03-17T19:07:03.6737780Z * [new branch] gh/seemethere/103/base -> origin/gh/seemethere/103/base 2022-03-17T19:07:03.6738851Z * [new branch] gh/seemethere/103/head -> origin/gh/seemethere/103/head 2022-03-17T19:07:03.6740206Z * [new branch] gh/seemethere/103/orig -> origin/gh/seemethere/103/orig 2022-03-17T19:07:03.6742658Z * [new branch] gh/seemethere/110/base -> origin/gh/seemethere/110/base 2022-03-17T19:07:03.6743835Z * [new branch] gh/seemethere/110/orig -> origin/gh/seemethere/110/orig 2022-03-17T19:07:03.6745846Z * [new branch] gh/seemethere/14/base -> origin/gh/seemethere/14/base 2022-03-17T19:07:03.6747523Z * [new branch] gh/seemethere/14/orig -> origin/gh/seemethere/14/orig 2022-03-17T19:07:03.6749338Z * [new branch] gh/seemethere/156/base -> origin/gh/seemethere/156/base 2022-03-17T19:07:03.6750513Z * [new branch] gh/seemethere/156/head -> origin/gh/seemethere/156/head 2022-03-17T19:07:03.6752158Z * [new branch] gh/seemethere/156/orig -> origin/gh/seemethere/156/orig 2022-03-17T19:07:03.6754061Z * [new branch] gh/seemethere/159/base -> origin/gh/seemethere/159/base 2022-03-17T19:07:03.6755335Z * [new branch] gh/seemethere/159/head -> origin/gh/seemethere/159/head 2022-03-17T19:07:03.6757015Z * [new branch] gh/seemethere/159/orig -> origin/gh/seemethere/159/orig 2022-03-17T19:07:03.6758803Z * [new branch] gh/seemethere/16/base -> origin/gh/seemethere/16/base 2022-03-17T19:07:03.6760412Z * [new branch] gh/seemethere/16/orig -> origin/gh/seemethere/16/orig 2022-03-17T19:07:03.6762051Z * [new branch] gh/seemethere/160/base -> origin/gh/seemethere/160/base 2022-03-17T19:07:03.6763202Z * [new branch] gh/seemethere/160/head -> origin/gh/seemethere/160/head 2022-03-17T19:07:03.6765016Z * [new branch] gh/seemethere/160/orig -> origin/gh/seemethere/160/orig 2022-03-17T19:07:03.6766885Z * [new branch] gh/seemethere/188/base -> origin/gh/seemethere/188/base 2022-03-17T19:07:03.6768789Z * [new branch] gh/seemethere/188/head -> origin/gh/seemethere/188/head 2022-03-17T19:07:03.6770075Z * [new branch] gh/seemethere/188/orig -> origin/gh/seemethere/188/orig 2022-03-17T19:07:03.6772019Z * [new branch] gh/seemethere/191/base -> origin/gh/seemethere/191/base 2022-03-17T19:07:03.6773190Z * [new branch] gh/seemethere/191/head -> origin/gh/seemethere/191/head 2022-03-17T19:07:03.6774823Z * [new branch] gh/seemethere/191/orig -> origin/gh/seemethere/191/orig 2022-03-17T19:07:03.6776586Z * [new branch] gh/seemethere/196/base -> origin/gh/seemethere/196/base 2022-03-17T19:07:03.6777777Z * [new branch] gh/seemethere/196/head -> origin/gh/seemethere/196/head 2022-03-17T19:07:03.6779562Z * [new branch] gh/seemethere/196/orig -> origin/gh/seemethere/196/orig 2022-03-17T19:07:03.6781092Z * [new branch] gh/seemethere/197/base -> origin/gh/seemethere/197/base 2022-03-17T19:07:03.6782849Z * [new branch] gh/seemethere/197/head -> origin/gh/seemethere/197/head 2022-03-17T19:07:03.6784018Z * [new branch] gh/seemethere/197/orig -> origin/gh/seemethere/197/orig 2022-03-17T19:07:03.6786050Z * [new branch] gh/seemethere/200/base -> origin/gh/seemethere/200/base 2022-03-17T19:07:03.6787677Z * [new branch] gh/seemethere/200/head -> origin/gh/seemethere/200/head 2022-03-17T19:07:03.6788812Z * [new branch] gh/seemethere/200/orig -> origin/gh/seemethere/200/orig 2022-03-17T19:07:03.6790885Z * [new branch] gh/seemethere/201/base -> origin/gh/seemethere/201/base 2022-03-17T19:07:03.6792033Z * [new branch] gh/seemethere/201/head -> origin/gh/seemethere/201/head 2022-03-17T19:07:03.6807989Z * [new branch] gh/seemethere/201/orig -> origin/gh/seemethere/201/orig 2022-03-17T19:07:03.6808801Z * [new branch] gh/seemethere/207/base -> origin/gh/seemethere/207/base 2022-03-17T19:07:03.6809295Z * [new branch] gh/seemethere/207/head -> origin/gh/seemethere/207/head 2022-03-17T19:07:03.6809683Z * [new branch] gh/seemethere/207/orig -> origin/gh/seemethere/207/orig 2022-03-17T19:07:03.6809978Z * [new branch] gh/seemethere/208/base -> origin/gh/seemethere/208/base 2022-03-17T19:07:03.6810197Z * [new branch] gh/seemethere/208/head -> origin/gh/seemethere/208/head 2022-03-17T19:07:03.6810440Z * [new branch] gh/seemethere/208/orig -> origin/gh/seemethere/208/orig 2022-03-17T19:07:03.6810774Z * [new branch] gh/seemethere/211/base -> origin/gh/seemethere/211/base 2022-03-17T19:07:03.6811422Z * [new branch] gh/seemethere/211/head -> origin/gh/seemethere/211/head 2022-03-17T19:07:03.6811747Z * [new branch] gh/seemethere/211/orig -> origin/gh/seemethere/211/orig 2022-03-17T19:07:03.6812022Z * [new branch] gh/seemethere/215/base -> origin/gh/seemethere/215/base 2022-03-17T19:07:03.6812208Z * [new branch] gh/seemethere/215/orig -> origin/gh/seemethere/215/orig 2022-03-17T19:07:03.6813299Z * [new branch] gh/seemethere/33/base -> origin/gh/seemethere/33/base 2022-03-17T19:07:03.6814697Z * [new branch] gh/seemethere/33/orig -> origin/gh/seemethere/33/orig 2022-03-17T19:07:03.6816740Z * [new branch] gh/seemethere/47/base -> origin/gh/seemethere/47/base 2022-03-17T19:07:03.6817988Z * [new branch] gh/seemethere/47/orig -> origin/gh/seemethere/47/orig 2022-03-17T19:07:03.6820058Z * [new branch] gh/seemethere/8/base -> origin/gh/seemethere/8/base 2022-03-17T19:07:03.6821418Z * [new branch] gh/seemethere/8/orig -> origin/gh/seemethere/8/orig 2022-03-17T19:07:03.6823752Z * [new branch] gh/seemethere/83/base -> origin/gh/seemethere/83/base 2022-03-17T19:07:03.6824938Z * [new branch] gh/seemethere/83/head -> origin/gh/seemethere/83/head 2022-03-17T19:07:03.6826695Z * [new branch] gh/seemethere/83/orig -> origin/gh/seemethere/83/orig 2022-03-17T19:07:03.6828691Z * [new branch] gh/seemethere/90/base -> origin/gh/seemethere/90/base 2022-03-17T19:07:03.6829888Z * [new branch] gh/seemethere/90/head -> origin/gh/seemethere/90/head 2022-03-17T19:07:03.6831314Z * [new branch] gh/seemethere/90/orig -> origin/gh/seemethere/90/orig 2022-03-17T19:07:03.6833627Z * [new branch] gh/shunting314/13/base -> origin/gh/shunting314/13/base 2022-03-17T19:07:03.6835014Z * [new branch] gh/shunting314/13/head -> origin/gh/shunting314/13/head 2022-03-17T19:07:03.6836412Z * [new branch] gh/shunting314/14/base -> origin/gh/shunting314/14/base 2022-03-17T19:07:03.6837975Z * [new branch] gh/shunting314/14/head -> origin/gh/shunting314/14/head 2022-03-17T19:07:03.6839154Z * [new branch] gh/shunting314/14/orig -> origin/gh/shunting314/14/orig 2022-03-17T19:07:03.6841229Z * [new branch] gh/shunting314/16/base -> origin/gh/shunting314/16/base 2022-03-17T19:07:03.6842439Z * [new branch] gh/shunting314/16/head -> origin/gh/shunting314/16/head 2022-03-17T19:07:03.6844169Z * [new branch] gh/shunting314/16/orig -> origin/gh/shunting314/16/orig 2022-03-17T19:07:03.6846032Z * [new branch] gh/shunting314/17/base -> origin/gh/shunting314/17/base 2022-03-17T19:07:03.6847073Z * [new branch] gh/shunting314/17/head -> origin/gh/shunting314/17/head 2022-03-17T19:07:03.6848491Z * [new branch] gh/shunting314/17/orig -> origin/gh/shunting314/17/orig 2022-03-17T19:07:03.6850463Z * [new branch] gh/shunting314/18/base -> origin/gh/shunting314/18/base 2022-03-17T19:07:03.6852386Z * [new branch] gh/shunting314/18/head -> origin/gh/shunting314/18/head 2022-03-17T19:07:03.6853426Z * [new branch] gh/shunting314/18/orig -> origin/gh/shunting314/18/orig 2022-03-17T19:07:03.6855415Z * [new branch] gh/shunting314/6/base -> origin/gh/shunting314/6/base 2022-03-17T19:07:03.6857012Z * [new branch] gh/shunting314/6/head -> origin/gh/shunting314/6/head 2022-03-17T19:07:03.6858766Z * [new branch] gh/shunting314/9/base -> origin/gh/shunting314/9/base 2022-03-17T19:07:03.6859960Z * [new branch] gh/shunting314/9/head -> origin/gh/shunting314/9/head 2022-03-17T19:07:03.6862434Z * [new branch] gh/shunting314/9/orig -> origin/gh/shunting314/9/orig 2022-03-17T19:07:03.6864731Z * [new branch] gh/simpkins/1/base -> origin/gh/simpkins/1/base 2022-03-17T19:07:03.6866958Z * [new branch] gh/simpkins/1/orig -> origin/gh/simpkins/1/orig 2022-03-17T19:07:03.6868676Z * [new branch] gh/simpkins/2/base -> origin/gh/simpkins/2/base 2022-03-17T19:07:03.6870058Z * [new branch] gh/simpkins/2/head -> origin/gh/simpkins/2/head 2022-03-17T19:07:03.6871642Z * [new branch] gh/simpkins/2/orig -> origin/gh/simpkins/2/orig 2022-03-17T19:07:03.6874339Z * [new branch] gh/sinannasir/12/base -> origin/gh/sinannasir/12/base 2022-03-17T19:07:03.6875729Z * [new branch] gh/sinannasir/12/orig -> origin/gh/sinannasir/12/orig 2022-03-17T19:07:03.6877942Z * [new branch] gh/sinannasir/8/base -> origin/gh/sinannasir/8/base 2022-03-17T19:07:03.6879832Z * [new branch] gh/sinannasir/9/base -> origin/gh/sinannasir/9/base 2022-03-17T19:07:03.6881363Z * [new branch] gh/sinannasir/9/head -> origin/gh/sinannasir/9/head 2022-03-17T19:07:03.6883015Z * [new branch] gh/sinannasir/9/orig -> origin/gh/sinannasir/9/orig 2022-03-17T19:07:03.6885316Z * [new branch] gh/smessmer/124/base -> origin/gh/smessmer/124/base 2022-03-17T19:07:03.6886697Z * [new branch] gh/smessmer/124/orig -> origin/gh/smessmer/124/orig 2022-03-17T19:07:03.6888540Z * [new branch] gh/smessmer/131/base -> origin/gh/smessmer/131/base 2022-03-17T19:07:03.6889923Z * [new branch] gh/smessmer/131/head -> origin/gh/smessmer/131/head 2022-03-17T19:07:03.6891342Z * [new branch] gh/smessmer/131/orig -> origin/gh/smessmer/131/orig 2022-03-17T19:07:03.6893425Z * [new branch] gh/smessmer/139/base -> origin/gh/smessmer/139/base 2022-03-17T19:07:03.6894657Z * [new branch] gh/smessmer/139/head -> origin/gh/smessmer/139/head 2022-03-17T19:07:03.6896934Z * [new branch] gh/smessmer/139/orig -> origin/gh/smessmer/139/orig 2022-03-17T19:07:03.6898506Z * [new branch] gh/smessmer/141/base -> origin/gh/smessmer/141/base 2022-03-17T19:07:03.6899934Z * [new branch] gh/smessmer/141/head -> origin/gh/smessmer/141/head 2022-03-17T19:07:03.6901202Z * [new branch] gh/smessmer/141/orig -> origin/gh/smessmer/141/orig 2022-03-17T19:07:03.6903684Z * [new branch] gh/smessmer/15/base -> origin/gh/smessmer/15/base 2022-03-17T19:07:03.6905415Z * [new branch] gh/smessmer/15/head -> origin/gh/smessmer/15/head 2022-03-17T19:07:03.6907261Z * [new branch] gh/smessmer/15/orig -> origin/gh/smessmer/15/orig 2022-03-17T19:07:03.6909200Z * [new branch] gh/smessmer/157/base -> origin/gh/smessmer/157/base 2022-03-17T19:07:03.6910331Z * [new branch] gh/smessmer/157/head -> origin/gh/smessmer/157/head 2022-03-17T19:07:03.6911799Z * [new branch] gh/smessmer/157/orig -> origin/gh/smessmer/157/orig 2022-03-17T19:07:03.6913756Z * [new branch] gh/smessmer/158/base -> origin/gh/smessmer/158/base 2022-03-17T19:07:03.6914913Z * [new branch] gh/smessmer/158/head -> origin/gh/smessmer/158/head 2022-03-17T19:07:03.6916232Z * [new branch] gh/smessmer/158/orig -> origin/gh/smessmer/158/orig 2022-03-17T19:07:03.6918318Z * [new branch] gh/smessmer/159/base -> origin/gh/smessmer/159/base 2022-03-17T19:07:03.6919428Z * [new branch] gh/smessmer/159/head -> origin/gh/smessmer/159/head 2022-03-17T19:07:03.6920983Z * [new branch] gh/smessmer/159/orig -> origin/gh/smessmer/159/orig 2022-03-17T19:07:03.6922928Z * [new branch] gh/smessmer/160/base -> origin/gh/smessmer/160/base 2022-03-17T19:07:03.6924089Z * [new branch] gh/smessmer/160/head -> origin/gh/smessmer/160/head 2022-03-17T19:07:03.6925763Z * [new branch] gh/smessmer/160/orig -> origin/gh/smessmer/160/orig 2022-03-17T19:07:03.6927466Z * [new branch] gh/smessmer/161/base -> origin/gh/smessmer/161/base 2022-03-17T19:07:03.6928526Z * [new branch] gh/smessmer/161/head -> origin/gh/smessmer/161/head 2022-03-17T19:07:03.6930195Z * [new branch] gh/smessmer/161/orig -> origin/gh/smessmer/161/orig 2022-03-17T19:07:03.6931881Z * [new branch] gh/smessmer/162/orig -> origin/gh/smessmer/162/orig 2022-03-17T19:07:03.6934147Z * [new branch] gh/smessmer/164/base -> origin/gh/smessmer/164/base 2022-03-17T19:07:03.6935503Z * [new branch] gh/smessmer/164/head -> origin/gh/smessmer/164/head 2022-03-17T19:07:03.6937187Z * [new branch] gh/smessmer/164/orig -> origin/gh/smessmer/164/orig 2022-03-17T19:07:03.6938999Z * [new branch] gh/smessmer/175/base -> origin/gh/smessmer/175/base 2022-03-17T19:07:03.6940126Z * [new branch] gh/smessmer/175/orig -> origin/gh/smessmer/175/orig 2022-03-17T19:07:03.6942560Z * [new branch] gh/smessmer/181/base -> origin/gh/smessmer/181/base 2022-03-17T19:07:03.6943716Z * [new branch] gh/smessmer/181/head -> origin/gh/smessmer/181/head 2022-03-17T19:07:03.6945378Z * [new branch] gh/smessmer/181/orig -> origin/gh/smessmer/181/orig 2022-03-17T19:07:03.6947346Z * [new branch] gh/smessmer/21/base -> origin/gh/smessmer/21/base 2022-03-17T19:07:03.6948589Z * [new branch] gh/smessmer/21/head -> origin/gh/smessmer/21/head 2022-03-17T19:07:03.6950404Z * [new branch] gh/smessmer/21/orig -> origin/gh/smessmer/21/orig 2022-03-17T19:07:03.6952160Z * [new branch] gh/smessmer/210/base -> origin/gh/smessmer/210/base 2022-03-17T19:07:03.6953338Z * [new branch] gh/smessmer/210/head -> origin/gh/smessmer/210/head 2022-03-17T19:07:03.6955034Z * [new branch] gh/smessmer/210/orig -> origin/gh/smessmer/210/orig 2022-03-17T19:07:03.6956818Z * [new branch] gh/smessmer/227/base -> origin/gh/smessmer/227/base 2022-03-17T19:07:03.6957963Z * [new branch] gh/smessmer/227/head -> origin/gh/smessmer/227/head 2022-03-17T19:07:03.6959984Z * [new branch] gh/smessmer/242/base -> origin/gh/smessmer/242/base 2022-03-17T19:07:03.6961401Z * [new branch] gh/smessmer/242/head -> origin/gh/smessmer/242/head 2022-03-17T19:07:03.6962461Z * [new branch] gh/smessmer/242/orig -> origin/gh/smessmer/242/orig 2022-03-17T19:07:03.6964567Z * [new branch] gh/smessmer/243/base -> origin/gh/smessmer/243/base 2022-03-17T19:07:03.6966296Z * [new branch] gh/smessmer/243/head -> origin/gh/smessmer/243/head 2022-03-17T19:07:03.6967559Z * [new branch] gh/smessmer/243/orig -> origin/gh/smessmer/243/orig 2022-03-17T19:07:03.6969246Z * [new branch] gh/smessmer/244/base -> origin/gh/smessmer/244/base 2022-03-17T19:07:03.6970456Z * [new branch] gh/smessmer/244/head -> origin/gh/smessmer/244/head 2022-03-17T19:07:03.6972112Z * [new branch] gh/smessmer/244/orig -> origin/gh/smessmer/244/orig 2022-03-17T19:07:03.6974020Z * [new branch] gh/smessmer/263/base -> origin/gh/smessmer/263/base 2022-03-17T19:07:03.6975440Z * [new branch] gh/smessmer/263/head -> origin/gh/smessmer/263/head 2022-03-17T19:07:03.6977037Z * [new branch] gh/smessmer/263/orig -> origin/gh/smessmer/263/orig 2022-03-17T19:07:03.6979243Z * [new branch] gh/smessmer/282/base -> origin/gh/smessmer/282/base 2022-03-17T19:07:03.6980907Z * [new branch] gh/smessmer/282/orig -> origin/gh/smessmer/282/orig 2022-03-17T19:07:03.6982898Z * [new branch] gh/smessmer/294/base -> origin/gh/smessmer/294/base 2022-03-17T19:07:03.6984051Z * [new branch] gh/smessmer/294/orig -> origin/gh/smessmer/294/orig 2022-03-17T19:07:03.6986448Z * [new branch] gh/smessmer/296/base -> origin/gh/smessmer/296/base 2022-03-17T19:07:03.6988033Z * [new branch] gh/smessmer/296/head -> origin/gh/smessmer/296/head 2022-03-17T19:07:03.6989775Z * [new branch] gh/smessmer/296/orig -> origin/gh/smessmer/296/orig 2022-03-17T19:07:03.6991633Z * [new branch] gh/smessmer/297/base -> origin/gh/smessmer/297/base 2022-03-17T19:07:03.6992926Z * [new branch] gh/smessmer/297/head -> origin/gh/smessmer/297/head 2022-03-17T19:07:03.6994248Z * [new branch] gh/smessmer/297/orig -> origin/gh/smessmer/297/orig 2022-03-17T19:07:03.6996316Z * [new branch] gh/smessmer/298/base -> origin/gh/smessmer/298/base 2022-03-17T19:07:03.6997565Z * [new branch] gh/smessmer/298/head -> origin/gh/smessmer/298/head 2022-03-17T19:07:03.6999151Z * [new branch] gh/smessmer/298/orig -> origin/gh/smessmer/298/orig 2022-03-17T19:07:03.7000893Z * [new branch] gh/smessmer/30/base -> origin/gh/smessmer/30/base 2022-03-17T19:07:03.7002084Z * [new branch] gh/smessmer/30/orig -> origin/gh/smessmer/30/orig 2022-03-17T19:07:03.7004126Z * [new branch] gh/smessmer/307/base -> origin/gh/smessmer/307/base 2022-03-17T19:07:03.7005304Z * [new branch] gh/smessmer/307/head -> origin/gh/smessmer/307/head 2022-03-17T19:07:03.7007292Z * [new branch] gh/smessmer/307/orig -> origin/gh/smessmer/307/orig 2022-03-17T19:07:03.7009008Z * [new branch] gh/smessmer/310/base -> origin/gh/smessmer/310/base 2022-03-17T19:07:03.7010145Z * [new branch] gh/smessmer/310/head -> origin/gh/smessmer/310/head 2022-03-17T19:07:03.7011828Z * [new branch] gh/smessmer/310/orig -> origin/gh/smessmer/310/orig 2022-03-17T19:07:03.7013559Z * [new branch] gh/smessmer/34/base -> origin/gh/smessmer/34/base 2022-03-17T19:07:03.7015161Z * [new branch] gh/smessmer/34/orig -> origin/gh/smessmer/34/orig 2022-03-17T19:07:03.7017655Z * [new branch] gh/smessmer/36/orig -> origin/gh/smessmer/36/orig 2022-03-17T19:07:03.7019449Z * [new branch] gh/smessmer/39/base -> origin/gh/smessmer/39/base 2022-03-17T19:07:03.7020846Z * [new branch] gh/smessmer/39/orig -> origin/gh/smessmer/39/orig 2022-03-17T19:07:03.7022960Z * [new branch] gh/smessmer/56/base -> origin/gh/smessmer/56/base 2022-03-17T19:07:03.7024143Z * [new branch] gh/smessmer/56/orig -> origin/gh/smessmer/56/orig 2022-03-17T19:07:03.7026341Z * [new branch] gh/smessmer/61/base -> origin/gh/smessmer/61/base 2022-03-17T19:07:03.7027223Z * [new branch] gh/smessmer/61/head -> origin/gh/smessmer/61/head 2022-03-17T19:07:03.7028948Z * [new branch] gh/smessmer/61/orig -> origin/gh/smessmer/61/orig 2022-03-17T19:07:03.7030791Z * [new branch] gh/smessmer/68/base -> origin/gh/smessmer/68/base 2022-03-17T19:07:03.7032038Z * [new branch] gh/smessmer/68/head -> origin/gh/smessmer/68/head 2022-03-17T19:07:03.7033745Z * [new branch] gh/smessmer/68/orig -> origin/gh/smessmer/68/orig 2022-03-17T19:07:03.7035578Z * [new branch] gh/smessmer/7/base -> origin/gh/smessmer/7/base 2022-03-17T19:07:03.7036776Z * [new branch] gh/smessmer/7/head -> origin/gh/smessmer/7/head 2022-03-17T19:07:03.7038778Z * [new branch] gh/smessmer/72/head -> origin/gh/smessmer/72/head 2022-03-17T19:07:03.7040383Z * [new branch] gh/smessmer/72/orig -> origin/gh/smessmer/72/orig 2022-03-17T19:07:03.7041574Z * [new branch] gh/smessmer/8/base -> origin/gh/smessmer/8/base 2022-03-17T19:07:03.7043558Z * [new branch] gh/smessmer/81/base -> origin/gh/smessmer/81/base 2022-03-17T19:07:03.7045401Z * [new branch] gh/smessmer/81/orig -> origin/gh/smessmer/81/orig 2022-03-17T19:07:03.7047042Z * [new branch] gh/smessmer/92/base -> origin/gh/smessmer/92/base 2022-03-17T19:07:03.7048410Z * [new branch] gh/smessmer/92/orig -> origin/gh/smessmer/92/orig 2022-03-17T19:07:03.7050916Z * [new branch] gh/soulitzer/23/base -> origin/gh/soulitzer/23/base 2022-03-17T19:07:03.7052094Z * [new branch] gh/soulitzer/23/head -> origin/gh/soulitzer/23/head 2022-03-17T19:07:03.7053799Z * [new branch] gh/soulitzer/23/orig -> origin/gh/soulitzer/23/orig 2022-03-17T19:07:03.7055570Z * [new branch] gh/soulitzer/24/base -> origin/gh/soulitzer/24/base 2022-03-17T19:07:03.7057094Z * [new branch] gh/soulitzer/24/head -> origin/gh/soulitzer/24/head 2022-03-17T19:07:03.7058259Z * [new branch] gh/soulitzer/24/orig -> origin/gh/soulitzer/24/orig 2022-03-17T19:07:03.7060111Z * [new branch] gh/soulitzer/25/base -> origin/gh/soulitzer/25/base 2022-03-17T19:07:03.7062354Z * [new branch] gh/soulitzer/25/head -> origin/gh/soulitzer/25/head 2022-03-17T19:07:03.7063561Z * [new branch] gh/soulitzer/25/orig -> origin/gh/soulitzer/25/orig 2022-03-17T19:07:03.7065702Z * [new branch] gh/soulitzer/27/base -> origin/gh/soulitzer/27/base 2022-03-17T19:07:03.7066768Z * [new branch] gh/soulitzer/27/head -> origin/gh/soulitzer/27/head 2022-03-17T19:07:03.7068596Z * [new branch] gh/soulitzer/27/orig -> origin/gh/soulitzer/27/orig 2022-03-17T19:07:03.7070149Z * [new branch] gh/soulitzer/49/base -> origin/gh/soulitzer/49/base 2022-03-17T19:07:03.7071468Z * [new branch] gh/soulitzer/49/head -> origin/gh/soulitzer/49/head 2022-03-17T19:07:03.7072994Z * [new branch] gh/soulitzer/49/orig -> origin/gh/soulitzer/49/orig 2022-03-17T19:07:03.7074805Z * [new branch] gh/soulitzer/57/base -> origin/gh/soulitzer/57/base 2022-03-17T19:07:03.7076157Z * [new branch] gh/soulitzer/57/head -> origin/gh/soulitzer/57/head 2022-03-17T19:07:03.7077557Z * [new branch] gh/soulitzer/57/orig -> origin/gh/soulitzer/57/orig 2022-03-17T19:07:03.7079054Z * [new branch] gh/soulitzer/58/base -> origin/gh/soulitzer/58/base 2022-03-17T19:07:03.7080494Z * [new branch] gh/soulitzer/58/head -> origin/gh/soulitzer/58/head 2022-03-17T19:07:03.7081923Z * [new branch] gh/soulitzer/58/orig -> origin/gh/soulitzer/58/orig 2022-03-17T19:07:03.7084151Z * [new branch] gh/soulitzer/59/base -> origin/gh/soulitzer/59/base 2022-03-17T19:07:03.7085137Z * [new branch] gh/soulitzer/59/head -> origin/gh/soulitzer/59/head 2022-03-17T19:07:03.7086443Z * [new branch] gh/soulitzer/59/orig -> origin/gh/soulitzer/59/orig 2022-03-17T19:07:03.7088196Z * [new branch] gh/soulitzer/64/base -> origin/gh/soulitzer/64/base 2022-03-17T19:07:03.7089520Z * [new branch] gh/soulitzer/64/head -> origin/gh/soulitzer/64/head 2022-03-17T19:07:03.7091060Z * [new branch] gh/soulitzer/64/orig -> origin/gh/soulitzer/64/orig 2022-03-17T19:07:03.7093329Z * [new branch] gh/soulitzer/65/base -> origin/gh/soulitzer/65/base 2022-03-17T19:07:03.7094502Z * [new branch] gh/soulitzer/65/head -> origin/gh/soulitzer/65/head 2022-03-17T19:07:03.7095884Z * [new branch] gh/soulitzer/65/orig -> origin/gh/soulitzer/65/orig 2022-03-17T19:07:03.7098115Z * [new branch] gh/soulitzer/66/base -> origin/gh/soulitzer/66/base 2022-03-17T19:07:03.7099275Z * [new branch] gh/soulitzer/66/head -> origin/gh/soulitzer/66/head 2022-03-17T19:07:03.7100888Z * [new branch] gh/soulitzer/66/orig -> origin/gh/soulitzer/66/orig 2022-03-17T19:07:03.7103357Z * [new branch] gh/soulitzer/67/base -> origin/gh/soulitzer/67/base 2022-03-17T19:07:03.7104718Z * [new branch] gh/soulitzer/67/head -> origin/gh/soulitzer/67/head 2022-03-17T19:07:03.7106023Z * [new branch] gh/soulitzer/67/orig -> origin/gh/soulitzer/67/orig 2022-03-17T19:07:03.7108372Z * [new branch] gh/splitinfinity/105/base -> origin/gh/splitinfinity/105/base 2022-03-17T19:07:03.7109486Z * [new branch] gh/splitinfinity/105/head -> origin/gh/splitinfinity/105/head 2022-03-17T19:07:03.7110985Z * [new branch] gh/splitinfinity/105/orig -> origin/gh/splitinfinity/105/orig 2022-03-17T19:07:03.7113167Z * [new branch] gh/splitinfinity/114/base -> origin/gh/splitinfinity/114/base 2022-03-17T19:07:03.7114397Z * [new branch] gh/splitinfinity/114/head -> origin/gh/splitinfinity/114/head 2022-03-17T19:07:03.7115798Z * [new branch] gh/splitinfinity/114/orig -> origin/gh/splitinfinity/114/orig 2022-03-17T19:07:03.7117501Z * [new branch] gh/splitinfinity/116/base -> origin/gh/splitinfinity/116/base 2022-03-17T19:07:03.7119791Z * [new branch] gh/splitinfinity/116/head -> origin/gh/splitinfinity/116/head 2022-03-17T19:07:03.7120829Z * [new branch] gh/splitinfinity/116/orig -> origin/gh/splitinfinity/116/orig 2022-03-17T19:07:03.7123028Z * [new branch] gh/splitinfinity/12/base -> origin/gh/splitinfinity/12/base 2022-03-17T19:07:03.7124384Z * [new branch] gh/splitinfinity/12/head -> origin/gh/splitinfinity/12/head 2022-03-17T19:07:03.7125869Z * [new branch] gh/splitinfinity/12/orig -> origin/gh/splitinfinity/12/orig 2022-03-17T19:07:03.7128063Z * [new branch] gh/splitinfinity/125/base -> origin/gh/splitinfinity/125/base 2022-03-17T19:07:03.7129200Z * [new branch] gh/splitinfinity/125/head -> origin/gh/splitinfinity/125/head 2022-03-17T19:07:03.7130673Z * [new branch] gh/splitinfinity/125/orig -> origin/gh/splitinfinity/125/orig 2022-03-17T19:07:03.7132740Z * [new branch] gh/splitinfinity/126/base -> origin/gh/splitinfinity/126/base 2022-03-17T19:07:03.7133826Z * [new branch] gh/splitinfinity/126/head -> origin/gh/splitinfinity/126/head 2022-03-17T19:07:03.7135234Z * [new branch] gh/splitinfinity/126/orig -> origin/gh/splitinfinity/126/orig 2022-03-17T19:07:03.7137866Z * [new branch] gh/splitinfinity/130/base -> origin/gh/splitinfinity/130/base 2022-03-17T19:07:03.7139020Z * [new branch] gh/splitinfinity/130/head -> origin/gh/splitinfinity/130/head 2022-03-17T19:07:03.7140711Z * [new branch] gh/splitinfinity/130/orig -> origin/gh/splitinfinity/130/orig 2022-03-17T19:07:03.7143060Z * [new branch] gh/splitinfinity/136/base -> origin/gh/splitinfinity/136/base 2022-03-17T19:07:03.7144041Z * [new branch] gh/splitinfinity/136/head -> origin/gh/splitinfinity/136/head 2022-03-17T19:07:03.7146243Z * [new branch] gh/splitinfinity/136/orig -> origin/gh/splitinfinity/136/orig 2022-03-17T19:07:03.7147712Z * [new branch] gh/splitinfinity/137/base -> origin/gh/splitinfinity/137/base 2022-03-17T19:07:03.7149201Z * [new branch] gh/splitinfinity/137/head -> origin/gh/splitinfinity/137/head 2022-03-17T19:07:03.7150659Z * [new branch] gh/splitinfinity/137/orig -> origin/gh/splitinfinity/137/orig 2022-03-17T19:07:03.7153238Z * [new branch] gh/splitinfinity/144/base -> origin/gh/splitinfinity/144/base 2022-03-17T19:07:03.7154352Z * [new branch] gh/splitinfinity/144/head -> origin/gh/splitinfinity/144/head 2022-03-17T19:07:03.7155783Z * [new branch] gh/splitinfinity/144/orig -> origin/gh/splitinfinity/144/orig 2022-03-17T19:07:03.7157745Z * [new branch] gh/splitinfinity/155/base -> origin/gh/splitinfinity/155/base 2022-03-17T19:07:03.7159092Z * [new branch] gh/splitinfinity/155/head -> origin/gh/splitinfinity/155/head 2022-03-17T19:07:03.7160437Z * [new branch] gh/splitinfinity/155/orig -> origin/gh/splitinfinity/155/orig 2022-03-17T19:07:03.7163026Z * [new branch] gh/splitinfinity/156/base -> origin/gh/splitinfinity/156/base 2022-03-17T19:07:03.7164014Z * [new branch] gh/splitinfinity/156/head -> origin/gh/splitinfinity/156/head 2022-03-17T19:07:03.7165412Z * [new branch] gh/splitinfinity/156/orig -> origin/gh/splitinfinity/156/orig 2022-03-17T19:07:03.7167908Z * [new branch] gh/splitinfinity/163/base -> origin/gh/splitinfinity/163/base 2022-03-17T19:07:03.7169043Z * [new branch] gh/splitinfinity/163/head -> origin/gh/splitinfinity/163/head 2022-03-17T19:07:03.7170609Z * [new branch] gh/splitinfinity/163/orig -> origin/gh/splitinfinity/163/orig 2022-03-17T19:07:03.7172116Z * [new branch] gh/splitinfinity/23/base -> origin/gh/splitinfinity/23/base 2022-03-17T19:07:03.7173506Z * [new branch] gh/splitinfinity/23/orig -> origin/gh/splitinfinity/23/orig 2022-03-17T19:07:03.7175872Z * [new branch] gh/splitinfinity/24/base -> origin/gh/splitinfinity/24/base 2022-03-17T19:07:03.7176831Z * [new branch] gh/splitinfinity/24/orig -> origin/gh/splitinfinity/24/orig 2022-03-17T19:07:03.7179014Z * [new branch] gh/splitinfinity/33/base -> origin/gh/splitinfinity/33/base 2022-03-17T19:07:03.7180090Z * [new branch] gh/splitinfinity/33/head -> origin/gh/splitinfinity/33/head 2022-03-17T19:07:03.7181890Z * [new branch] gh/splitinfinity/33/orig -> origin/gh/splitinfinity/33/orig 2022-03-17T19:07:03.7183452Z * [new branch] gh/splitinfinity/60/base -> origin/gh/splitinfinity/60/base 2022-03-17T19:07:03.7185260Z * [new branch] gh/splitinfinity/61/base -> origin/gh/splitinfinity/61/base 2022-03-17T19:07:03.7186996Z * [new branch] gh/splitinfinity/67/base -> origin/gh/splitinfinity/67/base 2022-03-17T19:07:03.7188290Z * [new branch] gh/splitinfinity/67/head -> origin/gh/splitinfinity/67/head 2022-03-17T19:07:03.7190755Z * [new branch] gh/splitinfinity/67/orig -> origin/gh/splitinfinity/67/orig 2022-03-17T19:07:03.7192096Z * [new branch] gh/splitinfinity/7/base -> origin/gh/splitinfinity/7/base 2022-03-17T19:07:03.7193541Z * [new branch] gh/splitinfinity/7/head -> origin/gh/splitinfinity/7/head 2022-03-17T19:07:03.7195475Z * [new branch] gh/splitinfinity/7/orig -> origin/gh/splitinfinity/7/orig 2022-03-17T19:07:03.7196746Z * [new branch] gh/splitinfinity/8/base -> origin/gh/splitinfinity/8/base 2022-03-17T19:07:03.7198207Z * [new branch] gh/splitinfinity/8/head -> origin/gh/splitinfinity/8/head 2022-03-17T19:07:03.7199919Z * [new branch] gh/splitinfinity/8/orig -> origin/gh/splitinfinity/8/orig 2022-03-17T19:07:03.7201791Z * [new branch] gh/splitinfinity/81/base -> origin/gh/splitinfinity/81/base 2022-03-17T19:07:03.7203054Z * [new branch] gh/splitinfinity/81/head -> origin/gh/splitinfinity/81/head 2022-03-17T19:07:03.7204575Z * [new branch] gh/splitinfinity/81/orig -> origin/gh/splitinfinity/81/orig 2022-03-17T19:07:03.7206884Z * [new branch] gh/splitinfinity/86/base -> origin/gh/splitinfinity/86/base 2022-03-17T19:07:03.7208111Z * [new branch] gh/splitinfinity/86/head -> origin/gh/splitinfinity/86/head 2022-03-17T19:07:03.7209498Z * [new branch] gh/splitinfinity/86/orig -> origin/gh/splitinfinity/86/orig 2022-03-17T19:07:03.7211585Z * [new branch] gh/splitinfinity/93/base -> origin/gh/splitinfinity/93/base 2022-03-17T19:07:03.7212675Z * [new branch] gh/splitinfinity/93/head -> origin/gh/splitinfinity/93/head 2022-03-17T19:07:03.7215065Z * [new branch] gh/splitinfinity/93/orig -> origin/gh/splitinfinity/93/orig 2022-03-17T19:07:03.7217102Z * [new branch] gh/splitinfinity/99/base -> origin/gh/splitinfinity/99/base 2022-03-17T19:07:03.7218366Z * [new branch] gh/splitinfinity/99/head -> origin/gh/splitinfinity/99/head 2022-03-17T19:07:03.7220724Z * [new branch] gh/splitinfinity/99/orig -> origin/gh/splitinfinity/99/orig 2022-03-17T19:07:03.7223157Z * [new branch] gh/stngo98/1/base -> origin/gh/stngo98/1/base 2022-03-17T19:07:03.7224211Z * [new branch] gh/stngo98/1/head -> origin/gh/stngo98/1/head 2022-03-17T19:07:03.7226056Z * [new branch] gh/stngo98/1/orig -> origin/gh/stngo98/1/orig 2022-03-17T19:07:03.7227967Z * [new branch] gh/stngo98/2/base -> origin/gh/stngo98/2/base 2022-03-17T19:07:03.7229184Z * [new branch] gh/stngo98/2/head -> origin/gh/stngo98/2/head 2022-03-17T19:07:03.7230440Z * [new branch] gh/stngo98/2/orig -> origin/gh/stngo98/2/orig 2022-03-17T19:07:03.7232794Z * [new branch] gh/suo/104/base -> origin/gh/suo/104/base 2022-03-17T19:07:03.7234380Z * [new branch] gh/suo/121/base -> origin/gh/suo/121/base 2022-03-17T19:07:03.7235671Z * [new branch] gh/suo/121/orig -> origin/gh/suo/121/orig 2022-03-17T19:07:03.7237607Z * [new branch] gh/suo/132/head -> origin/gh/suo/132/head 2022-03-17T19:07:03.7239448Z * [new branch] gh/suo/149/base -> origin/gh/suo/149/base 2022-03-17T19:07:03.7240684Z * [new branch] gh/suo/149/orig -> origin/gh/suo/149/orig 2022-03-17T19:07:03.7242714Z * [new branch] gh/suo/152/base -> origin/gh/suo/152/base 2022-03-17T19:07:03.7244098Z * [new branch] gh/suo/152/head -> origin/gh/suo/152/head 2022-03-17T19:07:03.7245573Z * [new branch] gh/suo/152/orig -> origin/gh/suo/152/orig 2022-03-17T19:07:03.7247559Z * [new branch] gh/suo/157/base -> origin/gh/suo/157/base 2022-03-17T19:07:03.7248773Z * [new branch] gh/suo/157/head -> origin/gh/suo/157/head 2022-03-17T19:07:03.7250700Z * [new branch] gh/suo/159/base -> origin/gh/suo/159/base 2022-03-17T19:07:03.7251793Z * [new branch] gh/suo/159/orig -> origin/gh/suo/159/orig 2022-03-17T19:07:03.7254022Z * [new branch] gh/suo/166/base -> origin/gh/suo/166/base 2022-03-17T19:07:03.7255329Z * [new branch] gh/suo/166/orig -> origin/gh/suo/166/orig 2022-03-17T19:07:03.7257470Z * [new branch] gh/suo/169/base -> origin/gh/suo/169/base 2022-03-17T19:07:03.7258701Z * [new branch] gh/suo/169/orig -> origin/gh/suo/169/orig 2022-03-17T19:07:03.7260732Z * [new branch] gh/suo/172/base -> origin/gh/suo/172/base 2022-03-17T19:07:03.7262174Z * [new branch] gh/suo/172/orig -> origin/gh/suo/172/orig 2022-03-17T19:07:03.7264488Z * [new branch] gh/suo/181/base -> origin/gh/suo/181/base 2022-03-17T19:07:03.7265687Z * [new branch] gh/suo/181/orig -> origin/gh/suo/181/orig 2022-03-17T19:07:03.7267916Z * [new branch] gh/suo/193/base -> origin/gh/suo/193/base 2022-03-17T19:07:03.7268925Z * [new branch] gh/suo/193/head -> origin/gh/suo/193/head 2022-03-17T19:07:03.7270306Z * [new branch] gh/suo/193/orig -> origin/gh/suo/193/orig 2022-03-17T19:07:03.7272463Z * [new branch] gh/suo/201/base -> origin/gh/suo/201/base 2022-03-17T19:07:03.7273603Z * [new branch] gh/suo/201/head -> origin/gh/suo/201/head 2022-03-17T19:07:03.7275666Z * [new branch] gh/suo/210/base -> origin/gh/suo/210/base 2022-03-17T19:07:03.7276786Z * [new branch] gh/suo/210/head -> origin/gh/suo/210/head 2022-03-17T19:07:03.7278220Z * [new branch] gh/suo/210/orig -> origin/gh/suo/210/orig 2022-03-17T19:07:03.7280327Z * [new branch] gh/suo/220/head -> origin/gh/suo/220/head 2022-03-17T19:07:03.7281466Z * [new branch] gh/suo/220/orig -> origin/gh/suo/220/orig 2022-03-17T19:07:03.7283643Z * [new branch] gh/suo/227/base -> origin/gh/suo/227/base 2022-03-17T19:07:03.7284788Z * [new branch] gh/suo/227/head -> origin/gh/suo/227/head 2022-03-17T19:07:03.7286254Z * [new branch] gh/suo/227/orig -> origin/gh/suo/227/orig 2022-03-17T19:07:03.7288385Z * [new branch] gh/suo/234/base -> origin/gh/suo/234/base 2022-03-17T19:07:03.7289966Z * [new branch] gh/suo/260/base -> origin/gh/suo/260/base 2022-03-17T19:07:03.7291283Z * [new branch] gh/suo/260/head -> origin/gh/suo/260/head 2022-03-17T19:07:03.7293491Z * [new branch] gh/suo/260/orig -> origin/gh/suo/260/orig 2022-03-17T19:07:03.7294986Z * [new branch] gh/suo/291/base -> origin/gh/suo/291/base 2022-03-17T19:07:03.7296985Z * [new branch] gh/suo/291/orig -> origin/gh/suo/291/orig 2022-03-17T19:07:03.7298307Z * [new branch] gh/suo/301/base -> origin/gh/suo/301/base 2022-03-17T19:07:03.7299712Z * [new branch] gh/suo/301/head -> origin/gh/suo/301/head 2022-03-17T19:07:03.7301092Z * [new branch] gh/suo/301/orig -> origin/gh/suo/301/orig 2022-03-17T19:07:03.7304547Z * [new branch] gh/suo/332/base -> origin/gh/suo/332/base 2022-03-17T19:07:03.7305992Z * [new branch] gh/suo/332/head -> origin/gh/suo/332/head 2022-03-17T19:07:03.7307556Z * [new branch] gh/suo/332/orig -> origin/gh/suo/332/orig 2022-03-17T19:07:03.7309449Z * [new branch] gh/suo/342/base -> origin/gh/suo/342/base 2022-03-17T19:07:03.7310694Z * [new branch] gh/suo/342/head -> origin/gh/suo/342/head 2022-03-17T19:07:03.7312076Z * [new branch] gh/suo/342/orig -> origin/gh/suo/342/orig 2022-03-17T19:07:03.7313936Z * [new branch] gh/suo/351/base -> origin/gh/suo/351/base 2022-03-17T19:07:03.7314998Z * [new branch] gh/suo/351/head -> origin/gh/suo/351/head 2022-03-17T19:07:03.7316386Z * [new branch] gh/suo/351/orig -> origin/gh/suo/351/orig 2022-03-17T19:07:03.7318594Z * [new branch] gh/suo/362/base -> origin/gh/suo/362/base 2022-03-17T19:07:03.7319757Z * [new branch] gh/suo/362/head -> origin/gh/suo/362/head 2022-03-17T19:07:03.7321224Z * [new branch] gh/suo/362/orig -> origin/gh/suo/362/orig 2022-03-17T19:07:03.7323540Z * [new branch] gh/suo/373/base -> origin/gh/suo/373/base 2022-03-17T19:07:03.7325375Z * [new branch] gh/suo/373/head -> origin/gh/suo/373/head 2022-03-17T19:07:03.7326531Z * [new branch] gh/suo/373/orig -> origin/gh/suo/373/orig 2022-03-17T19:07:03.7328160Z * [new branch] gh/suo/375/base -> origin/gh/suo/375/base 2022-03-17T19:07:03.7329324Z * [new branch] gh/suo/375/head -> origin/gh/suo/375/head 2022-03-17T19:07:03.7330492Z * [new branch] gh/suo/375/orig -> origin/gh/suo/375/orig 2022-03-17T19:07:03.7332557Z * [new branch] gh/suo/377/base -> origin/gh/suo/377/base 2022-03-17T19:07:03.7333830Z * [new branch] gh/suo/377/head -> origin/gh/suo/377/head 2022-03-17T19:07:03.7335705Z * [new branch] gh/suo/377/orig -> origin/gh/suo/377/orig 2022-03-17T19:07:03.7337638Z * [new branch] gh/suo/385/base -> origin/gh/suo/385/base 2022-03-17T19:07:03.7338948Z * [new branch] gh/suo/385/head -> origin/gh/suo/385/head 2022-03-17T19:07:03.7340815Z * [new branch] gh/suo/385/orig -> origin/gh/suo/385/orig 2022-03-17T19:07:03.7342843Z * [new branch] gh/suo/386/base -> origin/gh/suo/386/base 2022-03-17T19:07:03.7344074Z * [new branch] gh/suo/386/orig -> origin/gh/suo/386/orig 2022-03-17T19:07:03.7346168Z * [new branch] gh/suo/390/base -> origin/gh/suo/390/base 2022-03-17T19:07:03.7347175Z * [new branch] gh/suo/390/orig -> origin/gh/suo/390/orig 2022-03-17T19:07:03.7349783Z * [new branch] gh/suo/395/base -> origin/gh/suo/395/base 2022-03-17T19:07:03.7351928Z * [new branch] gh/suo/395/head -> origin/gh/suo/395/head 2022-03-17T19:07:03.7353035Z * [new branch] gh/suo/395/orig -> origin/gh/suo/395/orig 2022-03-17T19:07:03.7355185Z * [new branch] gh/suo/403/base -> origin/gh/suo/403/base 2022-03-17T19:07:03.7356367Z * [new branch] gh/suo/403/orig -> origin/gh/suo/403/orig 2022-03-17T19:07:03.7358616Z * [new branch] gh/suo/415/base -> origin/gh/suo/415/base 2022-03-17T19:07:03.7359573Z * [new branch] gh/suo/415/head -> origin/gh/suo/415/head 2022-03-17T19:07:03.7361238Z * [new branch] gh/suo/415/orig -> origin/gh/suo/415/orig 2022-03-17T19:07:03.7363118Z * [new branch] gh/suo/421/base -> origin/gh/suo/421/base 2022-03-17T19:07:03.7364729Z * [new branch] gh/suo/421/head -> origin/gh/suo/421/head 2022-03-17T19:07:03.7366217Z * [new branch] gh/suo/421/orig -> origin/gh/suo/421/orig 2022-03-17T19:07:03.7368308Z * [new branch] gh/suo/435/base -> origin/gh/suo/435/base 2022-03-17T19:07:03.7369484Z * [new branch] gh/suo/435/head -> origin/gh/suo/435/head 2022-03-17T19:07:03.7370829Z * [new branch] gh/suo/435/orig -> origin/gh/suo/435/orig 2022-03-17T19:07:03.7372965Z * [new branch] gh/suo/445/base -> origin/gh/suo/445/base 2022-03-17T19:07:03.7374144Z * [new branch] gh/suo/445/head -> origin/gh/suo/445/head 2022-03-17T19:07:03.7375504Z * [new branch] gh/suo/445/orig -> origin/gh/suo/445/orig 2022-03-17T19:07:03.7377636Z * [new branch] gh/suo/446/base -> origin/gh/suo/446/base 2022-03-17T19:07:03.7378821Z * [new branch] gh/suo/446/head -> origin/gh/suo/446/head 2022-03-17T19:07:03.7380194Z * [new branch] gh/suo/446/orig -> origin/gh/suo/446/orig 2022-03-17T19:07:03.7382753Z * [new branch] gh/suo/454/base -> origin/gh/suo/454/base 2022-03-17T19:07:03.7383924Z * [new branch] gh/suo/454/head -> origin/gh/suo/454/head 2022-03-17T19:07:03.7385781Z * [new branch] gh/suo/454/orig -> origin/gh/suo/454/orig 2022-03-17T19:07:03.7387714Z * [new branch] gh/suo/457/base -> origin/gh/suo/457/base 2022-03-17T19:07:03.7388872Z * [new branch] gh/suo/457/head -> origin/gh/suo/457/head 2022-03-17T19:07:03.7390208Z * [new branch] gh/suo/457/orig -> origin/gh/suo/457/orig 2022-03-17T19:07:03.7392442Z * [new branch] gh/suo/463/base -> origin/gh/suo/463/base 2022-03-17T19:07:03.7393907Z * [new branch] gh/suo/463/head -> origin/gh/suo/463/head 2022-03-17T19:07:03.7395774Z * [new branch] gh/suo/463/orig -> origin/gh/suo/463/orig 2022-03-17T19:07:03.7397042Z * [new branch] gh/suo/464/base -> origin/gh/suo/464/base 2022-03-17T19:07:03.7398919Z * [new branch] gh/suo/464/head -> origin/gh/suo/464/head 2022-03-17T19:07:03.7400171Z * [new branch] gh/suo/464/orig -> origin/gh/suo/464/orig 2022-03-17T19:07:03.7402022Z * [new branch] gh/suo/465/base -> origin/gh/suo/465/base 2022-03-17T19:07:03.7402955Z * [new branch] gh/suo/465/head -> origin/gh/suo/465/head 2022-03-17T19:07:03.7404288Z * [new branch] gh/suo/465/orig -> origin/gh/suo/465/orig 2022-03-17T19:07:03.7406634Z * [new branch] gh/suo/469/base -> origin/gh/suo/469/base 2022-03-17T19:07:03.7407839Z * [new branch] gh/suo/469/head -> origin/gh/suo/469/head 2022-03-17T19:07:03.7409481Z * [new branch] gh/suo/469/orig -> origin/gh/suo/469/orig 2022-03-17T19:07:03.7411417Z * [new branch] gh/suo/470/base -> origin/gh/suo/470/base 2022-03-17T19:07:03.7412492Z * [new branch] gh/suo/470/head -> origin/gh/suo/470/head 2022-03-17T19:07:03.7413778Z * [new branch] gh/suo/470/orig -> origin/gh/suo/470/orig 2022-03-17T19:07:03.7415923Z * [new branch] gh/suo/471/base -> origin/gh/suo/471/base 2022-03-17T19:07:03.7417122Z * [new branch] gh/suo/471/head -> origin/gh/suo/471/head 2022-03-17T19:07:03.7418442Z * [new branch] gh/suo/471/orig -> origin/gh/suo/471/orig 2022-03-17T19:07:03.7420460Z * [new branch] gh/suo/473/base -> origin/gh/suo/473/base 2022-03-17T19:07:03.7421879Z * [new branch] gh/suo/473/head -> origin/gh/suo/473/head 2022-03-17T19:07:03.7423326Z * [new branch] gh/suo/473/orig -> origin/gh/suo/473/orig 2022-03-17T19:07:03.7426256Z * [new branch] gh/suo/474/base -> origin/gh/suo/474/base 2022-03-17T19:07:03.7427311Z * [new branch] gh/suo/474/orig -> origin/gh/suo/474/orig 2022-03-17T19:07:03.7429516Z * [new branch] gh/suo/475/base -> origin/gh/suo/475/base 2022-03-17T19:07:03.7430520Z * [new branch] gh/suo/475/head -> origin/gh/suo/475/head 2022-03-17T19:07:03.7432429Z * [new branch] gh/suo/475/orig -> origin/gh/suo/475/orig 2022-03-17T19:07:03.7434391Z * [new branch] gh/suo/476/base -> origin/gh/suo/476/base 2022-03-17T19:07:03.7435518Z * [new branch] gh/suo/476/head -> origin/gh/suo/476/head 2022-03-17T19:07:03.7437259Z * [new branch] gh/suo/476/orig -> origin/gh/suo/476/orig 2022-03-17T19:07:03.7439051Z * [new branch] gh/suo/483/base -> origin/gh/suo/483/base 2022-03-17T19:07:03.7440170Z * [new branch] gh/suo/483/head -> origin/gh/suo/483/head 2022-03-17T19:07:03.7441843Z * [new branch] gh/suo/483/orig -> origin/gh/suo/483/orig 2022-03-17T19:07:03.7444160Z * [new branch] gh/suo/484/base -> origin/gh/suo/484/base 2022-03-17T19:07:03.7444788Z * [new branch] gh/suo/484/head -> origin/gh/suo/484/head 2022-03-17T19:07:03.7446399Z * [new branch] gh/suo/484/orig -> origin/gh/suo/484/orig 2022-03-17T19:07:03.7448402Z * [new branch] gh/suo/485/base -> origin/gh/suo/485/base 2022-03-17T19:07:03.7449448Z * [new branch] gh/suo/485/head -> origin/gh/suo/485/head 2022-03-17T19:07:03.7450801Z * [new branch] gh/suo/485/orig -> origin/gh/suo/485/orig 2022-03-17T19:07:03.7453221Z * [new branch] gh/suo/486/base -> origin/gh/suo/486/base 2022-03-17T19:07:03.7454139Z * [new branch] gh/suo/486/orig -> origin/gh/suo/486/orig 2022-03-17T19:07:03.7456280Z * [new branch] gh/suo/487/base -> origin/gh/suo/487/base 2022-03-17T19:07:03.7457141Z * [new branch] gh/suo/487/orig -> origin/gh/suo/487/orig 2022-03-17T19:07:03.7459411Z * [new branch] gh/suo/491/base -> origin/gh/suo/491/base 2022-03-17T19:07:03.7460748Z * [new branch] gh/suo/491/orig -> origin/gh/suo/491/orig 2022-03-17T19:07:03.7462894Z * [new branch] gh/suo/492/base -> origin/gh/suo/492/base 2022-03-17T19:07:03.7463891Z * [new branch] gh/suo/492/orig -> origin/gh/suo/492/orig 2022-03-17T19:07:03.7466200Z * [new branch] gh/suo/493/base -> origin/gh/suo/493/base 2022-03-17T19:07:03.7467304Z * [new branch] gh/suo/493/head -> origin/gh/suo/493/head 2022-03-17T19:07:03.7468770Z * [new branch] gh/suo/493/orig -> origin/gh/suo/493/orig 2022-03-17T19:07:03.7471677Z * [new branch] gh/suo/494/base -> origin/gh/suo/494/base 2022-03-17T19:07:03.7472723Z * [new branch] gh/suo/494/head -> origin/gh/suo/494/head 2022-03-17T19:07:03.7474142Z * [new branch] gh/suo/494/orig -> origin/gh/suo/494/orig 2022-03-17T19:07:03.7476274Z * [new branch] gh/suo/495/base -> origin/gh/suo/495/base 2022-03-17T19:07:03.7477239Z * [new branch] gh/suo/495/head -> origin/gh/suo/495/head 2022-03-17T19:07:03.7478610Z * [new branch] gh/suo/495/orig -> origin/gh/suo/495/orig 2022-03-17T19:07:03.7480777Z * [new branch] gh/suo/496/base -> origin/gh/suo/496/base 2022-03-17T19:07:03.7481881Z * [new branch] gh/suo/496/head -> origin/gh/suo/496/head 2022-03-17T19:07:03.7483227Z * [new branch] gh/suo/496/orig -> origin/gh/suo/496/orig 2022-03-17T19:07:03.7486100Z * [new branch] gh/suo/497/base -> origin/gh/suo/497/base 2022-03-17T19:07:03.7487229Z * [new branch] gh/suo/497/head -> origin/gh/suo/497/head 2022-03-17T19:07:03.7488640Z * [new branch] gh/suo/497/orig -> origin/gh/suo/497/orig 2022-03-17T19:07:03.7490799Z * [new branch] gh/suo/498/base -> origin/gh/suo/498/base 2022-03-17T19:07:03.7492008Z * [new branch] gh/suo/498/head -> origin/gh/suo/498/head 2022-03-17T19:07:03.7493425Z * [new branch] gh/suo/498/orig -> origin/gh/suo/498/orig 2022-03-17T19:07:03.7495597Z * [new branch] gh/suo/65/base -> origin/gh/suo/65/base 2022-03-17T19:07:03.7497148Z * [new branch] gh/suo/93/base -> origin/gh/suo/93/base 2022-03-17T19:07:03.7498642Z * [new branch] gh/suo/93/head -> origin/gh/suo/93/head 2022-03-17T19:07:03.7501178Z * [new branch] gh/supriyar/122/base -> origin/gh/supriyar/122/base 2022-03-17T19:07:03.7503419Z * [new branch] gh/supriyar/122/head -> origin/gh/supriyar/122/head 2022-03-17T19:07:03.7504448Z * [new branch] gh/supriyar/122/orig -> origin/gh/supriyar/122/orig 2022-03-17T19:07:03.7506738Z * [new branch] gh/supriyar/131/base -> origin/gh/supriyar/131/base 2022-03-17T19:07:03.7507906Z * [new branch] gh/supriyar/131/head -> origin/gh/supriyar/131/head 2022-03-17T19:07:03.7509345Z * [new branch] gh/supriyar/131/orig -> origin/gh/supriyar/131/orig 2022-03-17T19:07:03.7511901Z * [new branch] gh/supriyar/155/base -> origin/gh/supriyar/155/base 2022-03-17T19:07:03.7513131Z * [new branch] gh/supriyar/155/head -> origin/gh/supriyar/155/head 2022-03-17T19:07:03.7514614Z * [new branch] gh/supriyar/155/orig -> origin/gh/supriyar/155/orig 2022-03-17T19:07:03.7516765Z * [new branch] gh/supriyar/182/base -> origin/gh/supriyar/182/base 2022-03-17T19:07:03.7517858Z * [new branch] gh/supriyar/182/head -> origin/gh/supriyar/182/head 2022-03-17T19:07:03.7519890Z * [new branch] gh/supriyar/197/base -> origin/gh/supriyar/197/base 2022-03-17T19:07:03.7520992Z * [new branch] gh/supriyar/197/head -> origin/gh/supriyar/197/head 2022-03-17T19:07:03.7522437Z * [new branch] gh/supriyar/197/orig -> origin/gh/supriyar/197/orig 2022-03-17T19:07:03.7523876Z * [new branch] gh/supriyar/199/base -> origin/gh/supriyar/199/base 2022-03-17T19:07:03.7526434Z * [new branch] gh/supriyar/199/head -> origin/gh/supriyar/199/head 2022-03-17T19:07:03.7527569Z * [new branch] gh/supriyar/199/orig -> origin/gh/supriyar/199/orig 2022-03-17T19:07:03.7529515Z * [new branch] gh/supriyar/2/base -> origin/gh/supriyar/2/base 2022-03-17T19:07:03.7531120Z * [new branch] gh/supriyar/202/base -> origin/gh/supriyar/202/base 2022-03-17T19:07:03.7532680Z * [new branch] gh/supriyar/202/head -> origin/gh/supriyar/202/head 2022-03-17T19:07:03.7534059Z * [new branch] gh/supriyar/202/orig -> origin/gh/supriyar/202/orig 2022-03-17T19:07:03.7535603Z * [new branch] gh/supriyar/210/base -> origin/gh/supriyar/210/base 2022-03-17T19:07:03.7537049Z * [new branch] gh/supriyar/210/head -> origin/gh/supriyar/210/head 2022-03-17T19:07:03.7538467Z * [new branch] gh/supriyar/210/orig -> origin/gh/supriyar/210/orig 2022-03-17T19:07:03.7540228Z * [new branch] gh/supriyar/214/base -> origin/gh/supriyar/214/base 2022-03-17T19:07:03.7541854Z * [new branch] gh/supriyar/214/head -> origin/gh/supriyar/214/head 2022-03-17T19:07:03.7543244Z * [new branch] gh/supriyar/214/orig -> origin/gh/supriyar/214/orig 2022-03-17T19:07:03.7545462Z * [new branch] gh/supriyar/225/base -> origin/gh/supriyar/225/base 2022-03-17T19:07:03.7547156Z * [new branch] gh/supriyar/225/head -> origin/gh/supriyar/225/head 2022-03-17T19:07:03.7548609Z * [new branch] gh/supriyar/225/orig -> origin/gh/supriyar/225/orig 2022-03-17T19:07:03.7550260Z * [new branch] gh/supriyar/226/base -> origin/gh/supriyar/226/base 2022-03-17T19:07:03.7551879Z * [new branch] gh/supriyar/226/head -> origin/gh/supriyar/226/head 2022-03-17T19:07:03.7552982Z * [new branch] gh/supriyar/226/orig -> origin/gh/supriyar/226/orig 2022-03-17T19:07:03.7555146Z * [new branch] gh/supriyar/253/base -> origin/gh/supriyar/253/base 2022-03-17T19:07:03.7556440Z * [new branch] gh/supriyar/253/head -> origin/gh/supriyar/253/head 2022-03-17T19:07:03.7557855Z * [new branch] gh/supriyar/253/orig -> origin/gh/supriyar/253/orig 2022-03-17T19:07:03.7559615Z * [new branch] gh/supriyar/256/base -> origin/gh/supriyar/256/base 2022-03-17T19:07:03.7561147Z * [new branch] gh/supriyar/256/head -> origin/gh/supriyar/256/head 2022-03-17T19:07:03.7562614Z * [new branch] gh/supriyar/256/orig -> origin/gh/supriyar/256/orig 2022-03-17T19:07:03.7564894Z * [new branch] gh/supriyar/257/base -> origin/gh/supriyar/257/base 2022-03-17T19:07:03.7566017Z * [new branch] gh/supriyar/257/head -> origin/gh/supriyar/257/head 2022-03-17T19:07:03.7567579Z * [new branch] gh/supriyar/257/orig -> origin/gh/supriyar/257/orig 2022-03-17T19:07:03.7569870Z * [new branch] gh/supriyar/27/base -> origin/gh/supriyar/27/base 2022-03-17T19:07:03.7570991Z * [new branch] gh/supriyar/27/orig -> origin/gh/supriyar/27/orig 2022-03-17T19:07:03.7573162Z * [new branch] gh/supriyar/38/base -> origin/gh/supriyar/38/base 2022-03-17T19:07:03.7574149Z * [new branch] gh/supriyar/38/head -> origin/gh/supriyar/38/head 2022-03-17T19:07:03.7575637Z * [new branch] gh/supriyar/38/orig -> origin/gh/supriyar/38/orig 2022-03-17T19:07:03.7577760Z * [new branch] gh/supriyar/47/base -> origin/gh/supriyar/47/base 2022-03-17T19:07:03.7578867Z * [new branch] gh/supriyar/47/head -> origin/gh/supriyar/47/head 2022-03-17T19:07:03.7580616Z * [new branch] gh/supriyar/47/orig -> origin/gh/supriyar/47/orig 2022-03-17T19:07:03.7582924Z * [new branch] gh/supriyar/49/base -> origin/gh/supriyar/49/base 2022-03-17T19:07:03.7584007Z * [new branch] gh/supriyar/49/head -> origin/gh/supriyar/49/head 2022-03-17T19:07:03.7585400Z * [new branch] gh/supriyar/49/orig -> origin/gh/supriyar/49/orig 2022-03-17T19:07:03.7587874Z * [new branch] gh/supriyar/67/base -> origin/gh/supriyar/67/base 2022-03-17T19:07:03.7589082Z * [new branch] gh/supriyar/67/head -> origin/gh/supriyar/67/head 2022-03-17T19:07:03.7590658Z * [new branch] gh/supriyar/67/orig -> origin/gh/supriyar/67/orig 2022-03-17T19:07:03.7592105Z * [new branch] gh/supriyar/68/base -> origin/gh/supriyar/68/base 2022-03-17T19:07:03.7593566Z * [new branch] gh/supriyar/68/head -> origin/gh/supriyar/68/head 2022-03-17T19:07:03.7594939Z * [new branch] gh/supriyar/68/orig -> origin/gh/supriyar/68/orig 2022-03-17T19:07:03.7597650Z * [new branch] gh/supriyar/70/base -> origin/gh/supriyar/70/base 2022-03-17T19:07:03.7598795Z * [new branch] gh/supriyar/70/head -> origin/gh/supriyar/70/head 2022-03-17T19:07:03.7600102Z * [new branch] gh/supriyar/70/orig -> origin/gh/supriyar/70/orig 2022-03-17T19:07:03.7602270Z * [new branch] gh/supriyar/72/base -> origin/gh/supriyar/72/base 2022-03-17T19:07:03.7603029Z * [new branch] gh/supriyar/72/head -> origin/gh/supriyar/72/head 2022-03-17T19:07:03.7604286Z * [new branch] gh/supriyar/72/orig -> origin/gh/supriyar/72/orig 2022-03-17T19:07:03.7606224Z * [new branch] gh/supriyar/74/base -> origin/gh/supriyar/74/base 2022-03-17T19:07:03.7607491Z * [new branch] gh/supriyar/74/head -> origin/gh/supriyar/74/head 2022-03-17T19:07:03.7609326Z * [new branch] gh/supriyar/74/orig -> origin/gh/supriyar/74/orig 2022-03-17T19:07:03.7610455Z * [new branch] gh/supriyar/75/base -> origin/gh/supriyar/75/base 2022-03-17T19:07:03.7611777Z * [new branch] gh/supriyar/75/head -> origin/gh/supriyar/75/head 2022-03-17T19:07:03.7613152Z * [new branch] gh/supriyar/75/orig -> origin/gh/supriyar/75/orig 2022-03-17T19:07:03.7615736Z * [new branch] gh/supriyar/79/base -> origin/gh/supriyar/79/base 2022-03-17T19:07:03.7616989Z * [new branch] gh/supriyar/79/head -> origin/gh/supriyar/79/head 2022-03-17T19:07:03.7618347Z * [new branch] gh/supriyar/79/orig -> origin/gh/supriyar/79/orig 2022-03-17T19:07:03.7620511Z * [new branch] gh/supriyar/99/base -> origin/gh/supriyar/99/base 2022-03-17T19:07:03.7621940Z * [new branch] gh/supriyar/99/head -> origin/gh/supriyar/99/head 2022-03-17T19:07:03.7623338Z * [new branch] gh/supriyar/99/orig -> origin/gh/supriyar/99/orig 2022-03-17T19:07:03.7625843Z * [new branch] gh/swolchok/1/base -> origin/gh/swolchok/1/base 2022-03-17T19:07:03.7626938Z * [new branch] gh/swolchok/1/head -> origin/gh/swolchok/1/head 2022-03-17T19:07:03.7629034Z * [new branch] gh/swolchok/107/base -> origin/gh/swolchok/107/base 2022-03-17T19:07:03.7630240Z * [new branch] gh/swolchok/107/head -> origin/gh/swolchok/107/head 2022-03-17T19:07:03.7632232Z * [new branch] gh/swolchok/118/base -> origin/gh/swolchok/118/base 2022-03-17T19:07:03.7633477Z * [new branch] gh/swolchok/118/head -> origin/gh/swolchok/118/head 2022-03-17T19:07:03.7635070Z * [new branch] gh/swolchok/119/base -> origin/gh/swolchok/119/base 2022-03-17T19:07:03.7636180Z * [new branch] gh/swolchok/119/head -> origin/gh/swolchok/119/head 2022-03-17T19:07:03.7638376Z * [new branch] gh/swolchok/12/base -> origin/gh/swolchok/12/base 2022-03-17T19:07:03.7639691Z * [new branch] gh/swolchok/12/head -> origin/gh/swolchok/12/head 2022-03-17T19:07:03.7641288Z * [new branch] gh/swolchok/12/orig -> origin/gh/swolchok/12/orig 2022-03-17T19:07:03.7642634Z * [new branch] gh/swolchok/120/base -> origin/gh/swolchok/120/base 2022-03-17T19:07:03.7644531Z * [new branch] gh/swolchok/120/head -> origin/gh/swolchok/120/head 2022-03-17T19:07:03.7645890Z * [new branch] gh/swolchok/121/base -> origin/gh/swolchok/121/base 2022-03-17T19:07:03.7647056Z * [new branch] gh/swolchok/121/head -> origin/gh/swolchok/121/head 2022-03-17T19:07:03.7649324Z * [new branch] gh/swolchok/122/base -> origin/gh/swolchok/122/base 2022-03-17T19:07:03.7650582Z * [new branch] gh/swolchok/122/head -> origin/gh/swolchok/122/head 2022-03-17T19:07:03.7651979Z * [new branch] gh/swolchok/123/base -> origin/gh/swolchok/123/base 2022-03-17T19:07:03.7653827Z * [new branch] gh/swolchok/123/head -> origin/gh/swolchok/123/head 2022-03-17T19:07:03.7655155Z * [new branch] gh/swolchok/124/base -> origin/gh/swolchok/124/base 2022-03-17T19:07:03.7656731Z * [new branch] gh/swolchok/124/head -> origin/gh/swolchok/124/head 2022-03-17T19:07:03.7658083Z * [new branch] gh/swolchok/124/orig -> origin/gh/swolchok/124/orig 2022-03-17T19:07:03.7659668Z * [new branch] gh/swolchok/138/base -> origin/gh/swolchok/138/base 2022-03-17T19:07:03.7661267Z * [new branch] gh/swolchok/138/orig -> origin/gh/swolchok/138/orig 2022-03-17T19:07:03.7663564Z * [new branch] gh/swolchok/139/base -> origin/gh/swolchok/139/base 2022-03-17T19:07:03.7664763Z * [new branch] gh/swolchok/139/head -> origin/gh/swolchok/139/head 2022-03-17T19:07:03.7666179Z * [new branch] gh/swolchok/139/orig -> origin/gh/swolchok/139/orig 2022-03-17T19:07:03.7668186Z * [new branch] gh/swolchok/170/base -> origin/gh/swolchok/170/base 2022-03-17T19:07:03.7669357Z * [new branch] gh/swolchok/170/orig -> origin/gh/swolchok/170/orig 2022-03-17T19:07:03.7671359Z * [new branch] gh/swolchok/183/base -> origin/gh/swolchok/183/base 2022-03-17T19:07:03.7672519Z * [new branch] gh/swolchok/183/head -> origin/gh/swolchok/183/head 2022-03-17T19:07:03.7673903Z * [new branch] gh/swolchok/183/orig -> origin/gh/swolchok/183/orig 2022-03-17T19:07:03.7675926Z * [new branch] gh/swolchok/201/base -> origin/gh/swolchok/201/base 2022-03-17T19:07:03.7677549Z * [new branch] gh/swolchok/201/head -> origin/gh/swolchok/201/head 2022-03-17T19:07:03.7679065Z * [new branch] gh/swolchok/201/orig -> origin/gh/swolchok/201/orig 2022-03-17T19:07:03.7680755Z * [new branch] gh/swolchok/203/base -> origin/gh/swolchok/203/base 2022-03-17T19:07:03.7682309Z * [new branch] gh/swolchok/203/head -> origin/gh/swolchok/203/head 2022-03-17T19:07:03.7683812Z * [new branch] gh/swolchok/203/orig -> origin/gh/swolchok/203/orig 2022-03-17T19:07:03.7685887Z * [new branch] gh/swolchok/246/base -> origin/gh/swolchok/246/base 2022-03-17T19:07:03.7687139Z * [new branch] gh/swolchok/246/head -> origin/gh/swolchok/246/head 2022-03-17T19:07:03.7688608Z * [new branch] gh/swolchok/246/orig -> origin/gh/swolchok/246/orig 2022-03-17T19:07:03.7690769Z * [new branch] gh/swolchok/247/base -> origin/gh/swolchok/247/base 2022-03-17T19:07:03.7691790Z * [new branch] gh/swolchok/247/head -> origin/gh/swolchok/247/head 2022-03-17T19:07:03.7693201Z * [new branch] gh/swolchok/247/orig -> origin/gh/swolchok/247/orig 2022-03-17T19:07:03.7695441Z * [new branch] gh/swolchok/248/base -> origin/gh/swolchok/248/base 2022-03-17T19:07:03.7696443Z * [new branch] gh/swolchok/248/head -> origin/gh/swolchok/248/head 2022-03-17T19:07:03.7697972Z * [new branch] gh/swolchok/248/orig -> origin/gh/swolchok/248/orig 2022-03-17T19:07:03.7699906Z * [new branch] gh/swolchok/250/base -> origin/gh/swolchok/250/base 2022-03-17T19:07:03.7701105Z * [new branch] gh/swolchok/250/head -> origin/gh/swolchok/250/head 2022-03-17T19:07:03.7704147Z * [new branch] gh/swolchok/250/orig -> origin/gh/swolchok/250/orig 2022-03-17T19:07:03.7705564Z * [new branch] gh/swolchok/257/base -> origin/gh/swolchok/257/base 2022-03-17T19:07:03.7706925Z * [new branch] gh/swolchok/257/head -> origin/gh/swolchok/257/head 2022-03-17T19:07:03.7708403Z * [new branch] gh/swolchok/257/orig -> origin/gh/swolchok/257/orig 2022-03-17T19:07:03.7710622Z * [new branch] gh/swolchok/261/base -> origin/gh/swolchok/261/base 2022-03-17T19:07:03.7711719Z * [new branch] gh/swolchok/261/head -> origin/gh/swolchok/261/head 2022-03-17T19:07:03.7713158Z * [new branch] gh/swolchok/261/orig -> origin/gh/swolchok/261/orig 2022-03-17T19:07:03.7715299Z * [new branch] gh/swolchok/281/base -> origin/gh/swolchok/281/base 2022-03-17T19:07:03.7716349Z * [new branch] gh/swolchok/281/head -> origin/gh/swolchok/281/head 2022-03-17T19:07:03.7717834Z * [new branch] gh/swolchok/281/orig -> origin/gh/swolchok/281/orig 2022-03-17T19:07:03.7719690Z * [new branch] gh/swolchok/282/base -> origin/gh/swolchok/282/base 2022-03-17T19:07:03.7721014Z * [new branch] gh/swolchok/282/head -> origin/gh/swolchok/282/head 2022-03-17T19:07:03.7723219Z * [new branch] gh/swolchok/283/base -> origin/gh/swolchok/283/base 2022-03-17T19:07:03.7724357Z * [new branch] gh/swolchok/283/head -> origin/gh/swolchok/283/head 2022-03-17T19:07:03.7726596Z * [new branch] gh/swolchok/29/base -> origin/gh/swolchok/29/base 2022-03-17T19:07:03.7727749Z * [new branch] gh/swolchok/29/head -> origin/gh/swolchok/29/head 2022-03-17T19:07:03.7729870Z * [new branch] gh/swolchok/298/base -> origin/gh/swolchok/298/base 2022-03-17T19:07:03.7730895Z * [new branch] gh/swolchok/298/head -> origin/gh/swolchok/298/head 2022-03-17T19:07:03.7732358Z * [new branch] gh/swolchok/298/orig -> origin/gh/swolchok/298/orig 2022-03-17T19:07:03.7734458Z * [new branch] gh/swolchok/300/base -> origin/gh/swolchok/300/base 2022-03-17T19:07:03.7735643Z * [new branch] gh/swolchok/300/head -> origin/gh/swolchok/300/head 2022-03-17T19:07:03.7736992Z * [new branch] gh/swolchok/300/orig -> origin/gh/swolchok/300/orig 2022-03-17T19:07:03.7738899Z * [new branch] gh/swolchok/322/base -> origin/gh/swolchok/322/base 2022-03-17T19:07:03.7740113Z * [new branch] gh/swolchok/322/head -> origin/gh/swolchok/322/head 2022-03-17T19:07:03.7741748Z * [new branch] gh/swolchok/322/orig -> origin/gh/swolchok/322/orig 2022-03-17T19:07:03.7743974Z * [new branch] gh/swolchok/331/base -> origin/gh/swolchok/331/base 2022-03-17T19:07:03.7745130Z * [new branch] gh/swolchok/331/head -> origin/gh/swolchok/331/head 2022-03-17T19:07:03.7746531Z * [new branch] gh/swolchok/331/orig -> origin/gh/swolchok/331/orig 2022-03-17T19:07:03.7749156Z * [new branch] gh/swolchok/339/base -> origin/gh/swolchok/339/base 2022-03-17T19:07:03.7750364Z * [new branch] gh/swolchok/339/head -> origin/gh/swolchok/339/head 2022-03-17T19:07:03.7751736Z * [new branch] gh/swolchok/339/orig -> origin/gh/swolchok/339/orig 2022-03-17T19:07:03.7754253Z * [new branch] gh/swolchok/347/base -> origin/gh/swolchok/347/base 2022-03-17T19:07:03.7755408Z * [new branch] gh/swolchok/347/head -> origin/gh/swolchok/347/head 2022-03-17T19:07:03.7756813Z * [new branch] gh/swolchok/347/orig -> origin/gh/swolchok/347/orig 2022-03-17T19:07:03.7758343Z * [new branch] gh/swolchok/348/base -> origin/gh/swolchok/348/base 2022-03-17T19:07:03.7760545Z * [new branch] gh/swolchok/348/head -> origin/gh/swolchok/348/head 2022-03-17T19:07:03.7761940Z * [new branch] gh/swolchok/349/base -> origin/gh/swolchok/349/base 2022-03-17T19:07:03.7763747Z * [new branch] gh/swolchok/349/head -> origin/gh/swolchok/349/head 2022-03-17T19:07:03.7766039Z * [new branch] gh/swolchok/350/base -> origin/gh/swolchok/350/base 2022-03-17T19:07:03.7767335Z * [new branch] gh/swolchok/350/head -> origin/gh/swolchok/350/head 2022-03-17T19:07:03.7768884Z * [new branch] gh/swolchok/351/base -> origin/gh/swolchok/351/base 2022-03-17T19:07:03.7770430Z * [new branch] gh/swolchok/351/head -> origin/gh/swolchok/351/head 2022-03-17T19:07:03.7772745Z * [new branch] gh/swolchok/352/base -> origin/gh/swolchok/352/base 2022-03-17T19:07:03.7774178Z * [new branch] gh/swolchok/352/head -> origin/gh/swolchok/352/head 2022-03-17T19:07:03.7775751Z * [new branch] gh/swolchok/353/base -> origin/gh/swolchok/353/base 2022-03-17T19:07:03.7777243Z * [new branch] gh/swolchok/353/head -> origin/gh/swolchok/353/head 2022-03-17T19:07:03.7779523Z * [new branch] gh/swolchok/36/base -> origin/gh/swolchok/36/base 2022-03-17T19:07:03.7780768Z * [new branch] gh/swolchok/36/head -> origin/gh/swolchok/36/head 2022-03-17T19:07:03.7783268Z * [new branch] gh/swolchok/379/base -> origin/gh/swolchok/379/base 2022-03-17T19:07:03.7784311Z * [new branch] gh/swolchok/379/head -> origin/gh/swolchok/379/head 2022-03-17T19:07:03.7785761Z * [new branch] gh/swolchok/379/orig -> origin/gh/swolchok/379/orig 2022-03-17T19:07:03.7787297Z * [new branch] gh/swolchok/38/base -> origin/gh/swolchok/38/base 2022-03-17T19:07:03.7788789Z * [new branch] gh/swolchok/38/head -> origin/gh/swolchok/38/head 2022-03-17T19:07:03.7790173Z * [new branch] gh/swolchok/38/orig -> origin/gh/swolchok/38/orig 2022-03-17T19:07:03.7792551Z * [new branch] gh/swolchok/387/base -> origin/gh/swolchok/387/base 2022-03-17T19:07:03.7793680Z * [new branch] gh/swolchok/387/head -> origin/gh/swolchok/387/head 2022-03-17T19:07:03.7795027Z * [new branch] gh/swolchok/387/orig -> origin/gh/swolchok/387/orig 2022-03-17T19:07:03.7796607Z * [new branch] gh/swolchok/39/base -> origin/gh/swolchok/39/base 2022-03-17T19:07:03.7798090Z * [new branch] gh/swolchok/39/head -> origin/gh/swolchok/39/head 2022-03-17T19:07:03.7799676Z * [new branch] gh/swolchok/39/orig -> origin/gh/swolchok/39/orig 2022-03-17T19:07:03.7801900Z * [new branch] gh/swolchok/398/base -> origin/gh/swolchok/398/base 2022-03-17T19:07:03.7803126Z * [new branch] gh/swolchok/398/head -> origin/gh/swolchok/398/head 2022-03-17T19:07:03.7804533Z * [new branch] gh/swolchok/398/orig -> origin/gh/swolchok/398/orig 2022-03-17T19:07:03.7806104Z * [new branch] gh/swolchok/40/base -> origin/gh/swolchok/40/base 2022-03-17T19:07:03.7807638Z * [new branch] gh/swolchok/40/head -> origin/gh/swolchok/40/head 2022-03-17T19:07:03.7809149Z * [new branch] gh/swolchok/40/orig -> origin/gh/swolchok/40/orig 2022-03-17T19:07:03.7811231Z * [new branch] gh/swolchok/421/base -> origin/gh/swolchok/421/base 2022-03-17T19:07:03.7812272Z * [new branch] gh/swolchok/421/head -> origin/gh/swolchok/421/head 2022-03-17T19:07:03.7813690Z * [new branch] gh/swolchok/421/orig -> origin/gh/swolchok/421/orig 2022-03-17T19:07:03.7815760Z * [new branch] gh/swolchok/425/base -> origin/gh/swolchok/425/base 2022-03-17T19:07:03.7817366Z * [new branch] gh/swolchok/425/head -> origin/gh/swolchok/425/head 2022-03-17T19:07:03.7819678Z * [new branch] gh/swolchok/427/base -> origin/gh/swolchok/427/base 2022-03-17T19:07:03.7820834Z * [new branch] gh/swolchok/427/head -> origin/gh/swolchok/427/head 2022-03-17T19:07:03.7822340Z * [new branch] gh/swolchok/427/orig -> origin/gh/swolchok/427/orig 2022-03-17T19:07:03.7824626Z * [new branch] gh/swolchok/441/base -> origin/gh/swolchok/441/base 2022-03-17T19:07:03.7825769Z * [new branch] gh/swolchok/441/head -> origin/gh/swolchok/441/head 2022-03-17T19:07:03.7827296Z * [new branch] gh/swolchok/441/orig -> origin/gh/swolchok/441/orig 2022-03-17T19:07:03.7829429Z * [new branch] gh/swolchok/442/base -> origin/gh/swolchok/442/base 2022-03-17T19:07:03.7830573Z * [new branch] gh/swolchok/442/head -> origin/gh/swolchok/442/head 2022-03-17T19:07:03.7832028Z * [new branch] gh/swolchok/442/orig -> origin/gh/swolchok/442/orig 2022-03-17T19:07:03.7834210Z * [new branch] gh/swolchok/454/base -> origin/gh/swolchok/454/base 2022-03-17T19:07:03.7835371Z * [new branch] gh/swolchok/454/head -> origin/gh/swolchok/454/head 2022-03-17T19:07:03.7836787Z * [new branch] gh/swolchok/454/orig -> origin/gh/swolchok/454/orig 2022-03-17T19:07:03.7838846Z * [new branch] gh/swolchok/455/base -> origin/gh/swolchok/455/base 2022-03-17T19:07:03.7840068Z * [new branch] gh/swolchok/455/head -> origin/gh/swolchok/455/head 2022-03-17T19:07:03.7841446Z * [new branch] gh/swolchok/455/orig -> origin/gh/swolchok/455/orig 2022-03-17T19:07:03.7843661Z * [new branch] gh/swolchok/457/base -> origin/gh/swolchok/457/base 2022-03-17T19:07:03.7844776Z * [new branch] gh/swolchok/457/head -> origin/gh/swolchok/457/head 2022-03-17T19:07:03.7846210Z * [new branch] gh/swolchok/457/orig -> origin/gh/swolchok/457/orig 2022-03-17T19:07:03.7848325Z * [new branch] gh/swolchok/458/base -> origin/gh/swolchok/458/base 2022-03-17T19:07:03.7849400Z * [new branch] gh/swolchok/458/head -> origin/gh/swolchok/458/head 2022-03-17T19:07:03.7850814Z * [new branch] gh/swolchok/458/orig -> origin/gh/swolchok/458/orig 2022-03-17T19:07:03.7853034Z * [new branch] gh/swolchok/461/base -> origin/gh/swolchok/461/base 2022-03-17T19:07:03.7853986Z * [new branch] gh/swolchok/461/head -> origin/gh/swolchok/461/head 2022-03-17T19:07:03.7855436Z * [new branch] gh/swolchok/461/orig -> origin/gh/swolchok/461/orig 2022-03-17T19:07:03.7857579Z * [new branch] gh/swolchok/462/base -> origin/gh/swolchok/462/base 2022-03-17T19:07:03.7858718Z * [new branch] gh/swolchok/462/head -> origin/gh/swolchok/462/head 2022-03-17T19:07:03.7860122Z * [new branch] gh/swolchok/462/orig -> origin/gh/swolchok/462/orig 2022-03-17T19:07:03.7862627Z * [new branch] gh/swolchok/464/base -> origin/gh/swolchok/464/base 2022-03-17T19:07:03.7863969Z * [new branch] gh/swolchok/464/head -> origin/gh/swolchok/464/head 2022-03-17T19:07:03.7865408Z * [new branch] gh/swolchok/464/orig -> origin/gh/swolchok/464/orig 2022-03-17T19:07:03.7867532Z * [new branch] gh/swolchok/468/base -> origin/gh/swolchok/468/base 2022-03-17T19:07:03.7868739Z * [new branch] gh/swolchok/468/head -> origin/gh/swolchok/468/head 2022-03-17T19:07:03.7870182Z * [new branch] gh/swolchok/468/orig -> origin/gh/swolchok/468/orig 2022-03-17T19:07:03.7872292Z * [new branch] gh/swolchok/469/base -> origin/gh/swolchok/469/base 2022-03-17T19:07:03.7873517Z * [new branch] gh/swolchok/469/head -> origin/gh/swolchok/469/head 2022-03-17T19:07:03.7875748Z * [new branch] gh/swolchok/469/orig -> origin/gh/swolchok/469/orig 2022-03-17T19:07:03.7877488Z * [new branch] gh/swolchok/470/base -> origin/gh/swolchok/470/base 2022-03-17T19:07:03.7878913Z * [new branch] gh/swolchok/470/head -> origin/gh/swolchok/470/head 2022-03-17T19:07:03.7880289Z * [new branch] gh/swolchok/470/orig -> origin/gh/swolchok/470/orig 2022-03-17T19:07:03.7882380Z * [new branch] gh/swolchok/471/base -> origin/gh/swolchok/471/base 2022-03-17T19:07:03.7883510Z * [new branch] gh/swolchok/471/head -> origin/gh/swolchok/471/head 2022-03-17T19:07:03.7885389Z * [new branch] gh/swolchok/471/orig -> origin/gh/swolchok/471/orig 2022-03-17T19:07:03.7886808Z * [new branch] gh/swolchok/59/base -> origin/gh/swolchok/59/base 2022-03-17T19:07:03.7888159Z * [new branch] gh/swolchok/59/head -> origin/gh/swolchok/59/head 2022-03-17T19:07:03.7890244Z * [new branch] gh/swolchok/65/base -> origin/gh/swolchok/65/base 2022-03-17T19:07:03.7891502Z * [new branch] gh/swolchok/65/head -> origin/gh/swolchok/65/head 2022-03-17T19:07:03.7892968Z * [new branch] gh/swolchok/65/orig -> origin/gh/swolchok/65/orig 2022-03-17T19:07:03.7895223Z * [new branch] gh/swolchok/69/base -> origin/gh/swolchok/69/base 2022-03-17T19:07:03.7896490Z * [new branch] gh/swolchok/69/head -> origin/gh/swolchok/69/head 2022-03-17T19:07:03.7898167Z * [new branch] gh/swolchok/69/orig -> origin/gh/swolchok/69/orig 2022-03-17T19:07:03.7900117Z * [new branch] gh/swolchok/7/base -> origin/gh/swolchok/7/base 2022-03-17T19:07:03.7901213Z * [new branch] gh/swolchok/7/head -> origin/gh/swolchok/7/head 2022-03-17T19:07:03.7903977Z * [new branch] gh/swolchok/70/base -> origin/gh/swolchok/70/base 2022-03-17T19:07:03.7905376Z * [new branch] gh/swolchok/70/head -> origin/gh/swolchok/70/head 2022-03-17T19:07:03.7906757Z * [new branch] gh/swolchok/70/orig -> origin/gh/swolchok/70/orig 2022-03-17T19:07:03.7908693Z * [new branch] gh/swolchok/71/base -> origin/gh/swolchok/71/base 2022-03-17T19:07:03.7909790Z * [new branch] gh/swolchok/71/head -> origin/gh/swolchok/71/head 2022-03-17T19:07:03.7911278Z * [new branch] gh/swolchok/71/orig -> origin/gh/swolchok/71/orig 2022-03-17T19:07:03.7913174Z * [new branch] gh/swolchok/77/base -> origin/gh/swolchok/77/base 2022-03-17T19:07:03.7914380Z * [new branch] gh/swolchok/77/head -> origin/gh/swolchok/77/head 2022-03-17T19:07:03.7916675Z * [new branch] gh/swolchok/77/orig -> origin/gh/swolchok/77/orig 2022-03-17T19:07:03.7917971Z * [new branch] gh/swolchok/8/base -> origin/gh/swolchok/8/base 2022-03-17T19:07:03.7919362Z * [new branch] gh/swolchok/8/head -> origin/gh/swolchok/8/head 2022-03-17T19:07:03.7921668Z * [new branch] gh/syed-ahmed/4/base -> origin/gh/syed-ahmed/4/base 2022-03-17T19:07:03.7924299Z * [new branch] gh/syed-ahmed/5/base -> origin/gh/syed-ahmed/5/base 2022-03-17T19:07:03.7925606Z * [new branch] gh/syed-ahmed/7/base -> origin/gh/syed-ahmed/7/base 2022-03-17T19:07:03.7928448Z * [new branch] gh/taewookoh/2/base -> origin/gh/taewookoh/2/base 2022-03-17T19:07:03.7929667Z * [new branch] gh/taewookoh/2/head -> origin/gh/taewookoh/2/head 2022-03-17T19:07:03.7932039Z * [new branch] gh/taivu1998/10/base -> origin/gh/taivu1998/10/base 2022-03-17T19:07:03.7934024Z * [new branch] gh/taivu1998/10/head -> origin/gh/taivu1998/10/head 2022-03-17T19:07:03.7935138Z * [new branch] gh/taivu1998/10/orig -> origin/gh/taivu1998/10/orig 2022-03-17T19:07:03.7936820Z * [new branch] gh/taivu1998/11/base -> origin/gh/taivu1998/11/base 2022-03-17T19:07:03.7938112Z * [new branch] gh/taivu1998/11/head -> origin/gh/taivu1998/11/head 2022-03-17T19:07:03.7939512Z * [new branch] gh/taivu1998/11/orig -> origin/gh/taivu1998/11/orig 2022-03-17T19:07:03.7942078Z * [new branch] gh/taivu1998/12/base -> origin/gh/taivu1998/12/base 2022-03-17T19:07:03.7943352Z * [new branch] gh/taivu1998/12/head -> origin/gh/taivu1998/12/head 2022-03-17T19:07:03.7945320Z * [new branch] gh/taivu1998/12/orig -> origin/gh/taivu1998/12/orig 2022-03-17T19:07:03.7946433Z * [new branch] gh/taivu1998/13/base -> origin/gh/taivu1998/13/base 2022-03-17T19:07:03.7947896Z * [new branch] gh/taivu1998/13/head -> origin/gh/taivu1998/13/head 2022-03-17T19:07:03.7949173Z * [new branch] gh/taivu1998/13/orig -> origin/gh/taivu1998/13/orig 2022-03-17T19:07:03.7951223Z * [new branch] gh/taivu1998/7/base -> origin/gh/taivu1998/7/base 2022-03-17T19:07:03.7952297Z * [new branch] gh/taivu1998/7/head -> origin/gh/taivu1998/7/head 2022-03-17T19:07:03.7953740Z * [new branch] gh/taivu1998/7/orig -> origin/gh/taivu1998/7/orig 2022-03-17T19:07:03.7955119Z * [new branch] gh/taivu1998/8/base -> origin/gh/taivu1998/8/base 2022-03-17T19:07:03.7956469Z * [new branch] gh/taivu1998/8/head -> origin/gh/taivu1998/8/head 2022-03-17T19:07:03.7957861Z * [new branch] gh/taivu1998/8/orig -> origin/gh/taivu1998/8/orig 2022-03-17T19:07:03.7959502Z * [new branch] gh/taivu1998/9/base -> origin/gh/taivu1998/9/base 2022-03-17T19:07:03.7960759Z * [new branch] gh/taivu1998/9/head -> origin/gh/taivu1998/9/head 2022-03-17T19:07:03.7962649Z * [new branch] gh/taivu1998/9/orig -> origin/gh/taivu1998/9/orig 2022-03-17T19:07:03.7964131Z * [new branch] gh/taylorrobie/autotune_prototype -> origin/gh/taylorrobie/autotune_prototype 2022-03-17T19:07:03.7965604Z * [new branch] gh/taylorrobie/benchmark_utils_dataclassification -> origin/gh/taylorrobie/benchmark_utils_dataclassification 2022-03-17T19:07:03.7966907Z * [new branch] gh/taylorrobie/callgrind_backtest -> origin/gh/taylorrobie/callgrind_backtest 2022-03-17T19:07:03.7968201Z * [new branch] gh/taylorrobie/callgrind_scribe2 -> origin/gh/taylorrobie/callgrind_scribe2 2022-03-17T19:07:03.7969407Z * [new branch] gh/taylorrobie/composable_timer -> origin/gh/taylorrobie/composable_timer 2022-03-17T19:07:03.7970795Z * [new branch] gh/taylorrobie/fuzzed_op_microbenchmarks -> origin/gh/taylorrobie/fuzzed_op_microbenchmarks 2022-03-17T19:07:03.7972092Z * [new branch] gh/taylorrobie/kernel_overhead -> origin/gh/taylorrobie/kernel_overhead 2022-03-17T19:07:03.7973470Z * [new branch] gh/taylorrobie/unify_index -> origin/gh/taylorrobie/unify_index 2022-03-17T19:07:03.7974792Z * [new branch] gh/taylorrobie/wall_backtest -> origin/gh/taylorrobie/wall_backtest 2022-03-17T19:07:03.7977173Z * [new branch] gh/terrychenism/12/base -> origin/gh/terrychenism/12/base 2022-03-17T19:07:03.7978325Z * [new branch] gh/terrychenism/12/head -> origin/gh/terrychenism/12/head 2022-03-17T19:07:03.7979753Z * [new branch] gh/terrychenism/12/orig -> origin/gh/terrychenism/12/orig 2022-03-17T19:07:03.7982261Z * [new branch] gh/terrychenism/13/base -> origin/gh/terrychenism/13/base 2022-03-17T19:07:03.7983295Z * [new branch] gh/terrychenism/13/head -> origin/gh/terrychenism/13/head 2022-03-17T19:07:03.7984990Z * [new branch] gh/terrychenism/13/orig -> origin/gh/terrychenism/13/orig 2022-03-17T19:07:03.7986591Z * [new branch] gh/terrychenism/20/base -> origin/gh/terrychenism/20/base 2022-03-17T19:07:03.7988270Z * [new branch] gh/terrychenism/20/head -> origin/gh/terrychenism/20/head 2022-03-17T19:07:03.7989600Z * [new branch] gh/terrychenism/20/orig -> origin/gh/terrychenism/20/orig 2022-03-17T19:07:03.7991633Z * [new branch] gh/terrychenism/21/base -> origin/gh/terrychenism/21/base 2022-03-17T19:07:03.7992667Z * [new branch] gh/terrychenism/21/head -> origin/gh/terrychenism/21/head 2022-03-17T19:07:03.7994321Z * [new branch] gh/terrychenism/21/orig -> origin/gh/terrychenism/21/orig 2022-03-17T19:07:03.7996272Z * [new branch] gh/terrychenism/22/base -> origin/gh/terrychenism/22/base 2022-03-17T19:07:03.7997463Z * [new branch] gh/terrychenism/22/head -> origin/gh/terrychenism/22/head 2022-03-17T19:07:03.7998836Z * [new branch] gh/terrychenism/22/orig -> origin/gh/terrychenism/22/orig 2022-03-17T19:07:03.8001345Z * [new branch] gh/tierex/1/base -> origin/gh/tierex/1/base 2022-03-17T19:07:03.8002610Z * [new branch] gh/tierex/1/head -> origin/gh/tierex/1/head 2022-03-17T19:07:03.8003964Z * [new branch] gh/tierex/1/orig -> origin/gh/tierex/1/orig 2022-03-17T19:07:03.8006113Z * [new branch] gh/tierex/4/base -> origin/gh/tierex/4/base 2022-03-17T19:07:03.8007190Z * [new branch] gh/tierex/4/head -> origin/gh/tierex/4/head 2022-03-17T19:07:03.8008614Z * [new branch] gh/tierex/4/orig -> origin/gh/tierex/4/orig 2022-03-17T19:07:03.8010658Z * [new branch] gh/tierex/5/base -> origin/gh/tierex/5/base 2022-03-17T19:07:03.8011770Z * [new branch] gh/tierex/5/head -> origin/gh/tierex/5/head 2022-03-17T19:07:03.8013121Z * [new branch] gh/tierex/5/orig -> origin/gh/tierex/5/orig 2022-03-17T19:07:03.8014738Z * [new branch] gh/tierex/6/base -> origin/gh/tierex/6/base 2022-03-17T19:07:03.8016036Z * [new branch] gh/tierex/6/head -> origin/gh/tierex/6/head 2022-03-17T19:07:03.8017770Z * [new branch] gh/tierex/6/orig -> origin/gh/tierex/6/orig 2022-03-17T19:07:03.8019839Z * [new branch] gh/tktrungna/14/base -> origin/gh/tktrungna/14/base 2022-03-17T19:07:03.8021103Z * [new branch] gh/tktrungna/14/head -> origin/gh/tktrungna/14/head 2022-03-17T19:07:03.8022795Z * [new branch] gh/tktrungna/14/orig -> origin/gh/tktrungna/14/orig 2022-03-17T19:07:03.8024137Z * [new branch] gh/tktrungna/15/base -> origin/gh/tktrungna/15/base 2022-03-17T19:07:03.8025536Z * [new branch] gh/tktrungna/15/head -> origin/gh/tktrungna/15/head 2022-03-17T19:07:03.8027223Z * [new branch] gh/tktrungna/15/orig -> origin/gh/tktrungna/15/orig 2022-03-17T19:07:03.8029145Z * [new branch] gh/tktrungna/16/base -> origin/gh/tktrungna/16/base 2022-03-17T19:07:03.8030383Z * [new branch] gh/tktrungna/16/head -> origin/gh/tktrungna/16/head 2022-03-17T19:07:03.8031753Z * [new branch] gh/tktrungna/16/orig -> origin/gh/tktrungna/16/orig 2022-03-17T19:07:03.8033815Z * [new branch] gh/tktrungna/17/base -> origin/gh/tktrungna/17/base 2022-03-17T19:07:03.8035946Z * [new branch] gh/tktrungna/17/head -> origin/gh/tktrungna/17/head 2022-03-17T19:07:03.8037123Z * [new branch] gh/tktrungna/17/orig -> origin/gh/tktrungna/17/orig 2022-03-17T19:07:03.8039325Z * [new branch] gh/tktrungna/18/base -> origin/gh/tktrungna/18/base 2022-03-17T19:07:03.8040465Z * [new branch] gh/tktrungna/18/head -> origin/gh/tktrungna/18/head 2022-03-17T19:07:03.8041843Z * [new branch] gh/tktrungna/18/orig -> origin/gh/tktrungna/18/orig 2022-03-17T19:07:03.8043957Z * [new branch] gh/tktrungna/19/base -> origin/gh/tktrungna/19/base 2022-03-17T19:07:03.8044961Z * [new branch] gh/tktrungna/19/head -> origin/gh/tktrungna/19/head 2022-03-17T19:07:03.8046717Z * [new branch] gh/tktrungna/19/orig -> origin/gh/tktrungna/19/orig 2022-03-17T19:07:03.8048755Z * [new branch] gh/tugsbayasgalan/1/base -> origin/gh/tugsbayasgalan/1/base 2022-03-17T19:07:03.8049870Z * [new branch] gh/tugsbayasgalan/1/head -> origin/gh/tugsbayasgalan/1/head 2022-03-17T19:07:03.8051885Z * [new branch] gh/tugsbayasgalan/10/base -> origin/gh/tugsbayasgalan/10/base 2022-03-17T19:07:03.8053046Z * [new branch] gh/tugsbayasgalan/10/head -> origin/gh/tugsbayasgalan/10/head 2022-03-17T19:07:03.8055180Z * [new branch] gh/tugsbayasgalan/16/base -> origin/gh/tugsbayasgalan/16/base 2022-03-17T19:07:03.8056371Z * [new branch] gh/tugsbayasgalan/16/head -> origin/gh/tugsbayasgalan/16/head 2022-03-17T19:07:03.8058201Z * [new branch] gh/tugsbayasgalan/16/orig -> origin/gh/tugsbayasgalan/16/orig 2022-03-17T19:07:03.8060113Z * [new branch] gh/tugsbayasgalan/19/base -> origin/gh/tugsbayasgalan/19/base 2022-03-17T19:07:03.8061201Z * [new branch] gh/tugsbayasgalan/19/head -> origin/gh/tugsbayasgalan/19/head 2022-03-17T19:07:03.8062861Z * [new branch] gh/tugsbayasgalan/19/orig -> origin/gh/tugsbayasgalan/19/orig 2022-03-17T19:07:03.8064722Z * [new branch] gh/tugsbayasgalan/20/base -> origin/gh/tugsbayasgalan/20/base 2022-03-17T19:07:03.8065785Z * [new branch] gh/tugsbayasgalan/20/head -> origin/gh/tugsbayasgalan/20/head 2022-03-17T19:07:03.8067149Z * [new branch] gh/tugsbayasgalan/20/orig -> origin/gh/tugsbayasgalan/20/orig 2022-03-17T19:07:03.8069132Z * [new branch] gh/tugsbayasgalan/36/base -> origin/gh/tugsbayasgalan/36/base 2022-03-17T19:07:03.8070264Z * [new branch] gh/tugsbayasgalan/36/head -> origin/gh/tugsbayasgalan/36/head 2022-03-17T19:07:03.8071654Z * [new branch] gh/tugsbayasgalan/36/orig -> origin/gh/tugsbayasgalan/36/orig 2022-03-17T19:07:03.8073641Z * [new branch] gh/tugsbayasgalan/37/base -> origin/gh/tugsbayasgalan/37/base 2022-03-17T19:07:03.8074810Z * [new branch] gh/tugsbayasgalan/37/head -> origin/gh/tugsbayasgalan/37/head 2022-03-17T19:07:03.8076178Z * [new branch] gh/tugsbayasgalan/37/orig -> origin/gh/tugsbayasgalan/37/orig 2022-03-17T19:07:03.8078290Z * [new branch] gh/tugsbayasgalan/42/base -> origin/gh/tugsbayasgalan/42/base 2022-03-17T19:07:03.8079463Z * [new branch] gh/tugsbayasgalan/42/head -> origin/gh/tugsbayasgalan/42/head 2022-03-17T19:07:03.8080828Z * [new branch] gh/tugsbayasgalan/42/orig -> origin/gh/tugsbayasgalan/42/orig 2022-03-17T19:07:03.8082820Z * [new branch] gh/tugsbayasgalan/47/base -> origin/gh/tugsbayasgalan/47/base 2022-03-17T19:07:03.8083923Z * [new branch] gh/tugsbayasgalan/47/head -> origin/gh/tugsbayasgalan/47/head 2022-03-17T19:07:03.8085294Z * [new branch] gh/tugsbayasgalan/47/orig -> origin/gh/tugsbayasgalan/47/orig 2022-03-17T19:07:03.8086920Z * [new branch] gh/tugsbayasgalan/48/base -> origin/gh/tugsbayasgalan/48/base 2022-03-17T19:07:03.8088247Z * [new branch] gh/tugsbayasgalan/48/head -> origin/gh/tugsbayasgalan/48/head 2022-03-17T19:07:03.8089623Z * [new branch] gh/tugsbayasgalan/48/orig -> origin/gh/tugsbayasgalan/48/orig 2022-03-17T19:07:03.8091452Z * [new branch] gh/tugsbayasgalan/49/base -> origin/gh/tugsbayasgalan/49/base 2022-03-17T19:07:03.8092545Z * [new branch] gh/tugsbayasgalan/49/head -> origin/gh/tugsbayasgalan/49/head 2022-03-17T19:07:03.8094592Z * [new branch] gh/tugsbayasgalan/52/base -> origin/gh/tugsbayasgalan/52/base 2022-03-17T19:07:03.8095964Z * [new branch] gh/tugsbayasgalan/52/head -> origin/gh/tugsbayasgalan/52/head 2022-03-17T19:07:03.8097373Z * [new branch] gh/tugsbayasgalan/52/orig -> origin/gh/tugsbayasgalan/52/orig 2022-03-17T19:07:03.8099888Z * [new branch] gh/tugsbayasgalan/53/base -> origin/gh/tugsbayasgalan/53/base 2022-03-17T19:07:03.8101343Z * [new branch] gh/tugsbayasgalan/53/head -> origin/gh/tugsbayasgalan/53/head 2022-03-17T19:07:03.8103529Z * [new branch] gh/tugsbayasgalan/53/orig -> origin/gh/tugsbayasgalan/53/orig 2022-03-17T19:07:03.8105298Z * [new branch] gh/tugsbayasgalan/54/base -> origin/gh/tugsbayasgalan/54/base 2022-03-17T19:07:03.8106366Z * [new branch] gh/tugsbayasgalan/54/head -> origin/gh/tugsbayasgalan/54/head 2022-03-17T19:07:03.8107733Z * [new branch] gh/tugsbayasgalan/54/orig -> origin/gh/tugsbayasgalan/54/orig 2022-03-17T19:07:03.8109810Z * [new branch] gh/tugsbayasgalan/55/base -> origin/gh/tugsbayasgalan/55/base 2022-03-17T19:07:03.8110951Z * [new branch] gh/tugsbayasgalan/55/head -> origin/gh/tugsbayasgalan/55/head 2022-03-17T19:07:03.8112360Z * [new branch] gh/tugsbayasgalan/55/orig -> origin/gh/tugsbayasgalan/55/orig 2022-03-17T19:07:03.8114508Z * [new branch] gh/tugsbayasgalan/56/base -> origin/gh/tugsbayasgalan/56/base 2022-03-17T19:07:03.8115581Z * [new branch] gh/tugsbayasgalan/56/head -> origin/gh/tugsbayasgalan/56/head 2022-03-17T19:07:03.8117500Z * [new branch] gh/tugsbayasgalan/56/orig -> origin/gh/tugsbayasgalan/56/orig 2022-03-17T19:07:03.8118929Z * [new branch] gh/tugsbayasgalan/57/base -> origin/gh/tugsbayasgalan/57/base 2022-03-17T19:07:03.8120582Z * [new branch] gh/tugsbayasgalan/57/head -> origin/gh/tugsbayasgalan/57/head 2022-03-17T19:07:03.8121884Z * [new branch] gh/tugsbayasgalan/57/orig -> origin/gh/tugsbayasgalan/57/orig 2022-03-17T19:07:03.8124016Z * [new branch] gh/tugsbayasgalan/58/base -> origin/gh/tugsbayasgalan/58/base 2022-03-17T19:07:03.8125160Z * [new branch] gh/tugsbayasgalan/58/head -> origin/gh/tugsbayasgalan/58/head 2022-03-17T19:07:03.8127214Z * [new branch] gh/tugsbayasgalan/59/base -> origin/gh/tugsbayasgalan/59/base 2022-03-17T19:07:03.8128213Z * [new branch] gh/tugsbayasgalan/59/head -> origin/gh/tugsbayasgalan/59/head 2022-03-17T19:07:03.8129482Z * [new branch] gh/tugsbayasgalan/59/orig -> origin/gh/tugsbayasgalan/59/orig 2022-03-17T19:07:03.8131683Z * [new branch] gh/tugsbayasgalan/60/base -> origin/gh/tugsbayasgalan/60/base 2022-03-17T19:07:03.8132737Z * [new branch] gh/tugsbayasgalan/60/head -> origin/gh/tugsbayasgalan/60/head 2022-03-17T19:07:03.8134132Z * [new branch] gh/tugsbayasgalan/60/orig -> origin/gh/tugsbayasgalan/60/orig 2022-03-17T19:07:03.8136393Z * [new branch] gh/tugsbayasgalan/61/base -> origin/gh/tugsbayasgalan/61/base 2022-03-17T19:07:03.8137558Z * [new branch] gh/tugsbayasgalan/61/head -> origin/gh/tugsbayasgalan/61/head 2022-03-17T19:07:03.8138934Z * [new branch] gh/tugsbayasgalan/61/orig -> origin/gh/tugsbayasgalan/61/orig 2022-03-17T19:07:03.8141010Z * [new branch] gh/tugsbayasgalan/62/base -> origin/gh/tugsbayasgalan/62/base 2022-03-17T19:07:03.8142536Z * [new branch] gh/tugsbayasgalan/62/head -> origin/gh/tugsbayasgalan/62/head 2022-03-17T19:07:03.8143881Z * [new branch] gh/tugsbayasgalan/62/orig -> origin/gh/tugsbayasgalan/62/orig 2022-03-17T19:07:03.8145893Z * [new branch] gh/tugsbayasgalan/63/base -> origin/gh/tugsbayasgalan/63/base 2022-03-17T19:07:03.8147033Z * [new branch] gh/tugsbayasgalan/63/head -> origin/gh/tugsbayasgalan/63/head 2022-03-17T19:07:03.8148330Z * [new branch] gh/tugsbayasgalan/63/orig -> origin/gh/tugsbayasgalan/63/orig 2022-03-17T19:07:03.8150413Z * [new branch] gh/tugsbayasgalan/64/base -> origin/gh/tugsbayasgalan/64/base 2022-03-17T19:07:03.8151557Z * [new branch] gh/tugsbayasgalan/64/head -> origin/gh/tugsbayasgalan/64/head 2022-03-17T19:07:03.8153013Z * [new branch] gh/tugsbayasgalan/64/orig -> origin/gh/tugsbayasgalan/64/orig 2022-03-17T19:07:03.8155030Z * [new branch] gh/tugsbayasgalan/65/base -> origin/gh/tugsbayasgalan/65/base 2022-03-17T19:07:03.8156999Z * [new branch] gh/tugsbayasgalan/65/head -> origin/gh/tugsbayasgalan/65/head 2022-03-17T19:07:03.8158130Z * [new branch] gh/tugsbayasgalan/65/orig -> origin/gh/tugsbayasgalan/65/orig 2022-03-17T19:07:03.8160262Z * [new branch] gh/tugsbayasgalan/67/base -> origin/gh/tugsbayasgalan/67/base 2022-03-17T19:07:03.8161452Z * [new branch] gh/tugsbayasgalan/67/head -> origin/gh/tugsbayasgalan/67/head 2022-03-17T19:07:03.8162863Z * [new branch] gh/tugsbayasgalan/67/orig -> origin/gh/tugsbayasgalan/67/orig 2022-03-17T19:07:03.8164958Z * [new branch] gh/tugsbayasgalan/68/base -> origin/gh/tugsbayasgalan/68/base 2022-03-17T19:07:03.8166129Z * [new branch] gh/tugsbayasgalan/68/head -> origin/gh/tugsbayasgalan/68/head 2022-03-17T19:07:03.8167549Z * [new branch] gh/tugsbayasgalan/68/orig -> origin/gh/tugsbayasgalan/68/orig 2022-03-17T19:07:03.8169645Z * [new branch] gh/tugsbayasgalan/71/base -> origin/gh/tugsbayasgalan/71/base 2022-03-17T19:07:03.8170683Z * [new branch] gh/tugsbayasgalan/71/head -> origin/gh/tugsbayasgalan/71/head 2022-03-17T19:07:03.8172038Z * [new branch] gh/tugsbayasgalan/71/orig -> origin/gh/tugsbayasgalan/71/orig 2022-03-17T19:07:03.8174062Z * [new branch] gh/tugsbayasgalan/72/base -> origin/gh/tugsbayasgalan/72/base 2022-03-17T19:07:03.8175206Z * [new branch] gh/tugsbayasgalan/72/head -> origin/gh/tugsbayasgalan/72/head 2022-03-17T19:07:03.8177061Z * [new branch] gh/tugsbayasgalan/72/orig -> origin/gh/tugsbayasgalan/72/orig 2022-03-17T19:07:03.8178614Z * [new branch] gh/tugsbayasgalan/77/base -> origin/gh/tugsbayasgalan/77/base 2022-03-17T19:07:03.8179896Z * [new branch] gh/tugsbayasgalan/77/head -> origin/gh/tugsbayasgalan/77/head 2022-03-17T19:07:03.8181485Z * [new branch] gh/tugsbayasgalan/77/orig -> origin/gh/tugsbayasgalan/77/orig 2022-03-17T19:07:03.8183559Z * [new branch] gh/tugsbayasgalan/78/base -> origin/gh/tugsbayasgalan/78/base 2022-03-17T19:07:03.8184625Z * [new branch] gh/tugsbayasgalan/78/head -> origin/gh/tugsbayasgalan/78/head 2022-03-17T19:07:03.8185964Z * [new branch] gh/tugsbayasgalan/78/orig -> origin/gh/tugsbayasgalan/78/orig 2022-03-17T19:07:03.8188026Z * [new branch] gh/tugsbayasgalan/82/base -> origin/gh/tugsbayasgalan/82/base 2022-03-17T19:07:03.8189103Z * [new branch] gh/tugsbayasgalan/82/head -> origin/gh/tugsbayasgalan/82/head 2022-03-17T19:07:03.8190483Z * [new branch] gh/tugsbayasgalan/82/orig -> origin/gh/tugsbayasgalan/82/orig 2022-03-17T19:07:03.8192522Z * [new branch] gh/tugsbayasgalan/85/base -> origin/gh/tugsbayasgalan/85/base 2022-03-17T19:07:03.8193735Z * [new branch] gh/tugsbayasgalan/85/head -> origin/gh/tugsbayasgalan/85/head 2022-03-17T19:07:03.8195154Z * [new branch] gh/tugsbayasgalan/85/orig -> origin/gh/tugsbayasgalan/85/orig 2022-03-17T19:07:03.8199470Z * [new branch] gh/tugsbayasgalan/87/base -> origin/gh/tugsbayasgalan/87/base 2022-03-17T19:07:03.8199903Z * [new branch] gh/tugsbayasgalan/87/head -> origin/gh/tugsbayasgalan/87/head 2022-03-17T19:07:03.8200390Z * [new branch] gh/tugsbayasgalan/87/orig -> origin/gh/tugsbayasgalan/87/orig 2022-03-17T19:07:03.8202393Z * [new branch] gh/tugsbayasgalan/88/base -> origin/gh/tugsbayasgalan/88/base 2022-03-17T19:07:03.8203530Z * [new branch] gh/tugsbayasgalan/88/head -> origin/gh/tugsbayasgalan/88/head 2022-03-17T19:07:03.8205184Z * [new branch] gh/tugsbayasgalan/88/orig -> origin/gh/tugsbayasgalan/88/orig 2022-03-17T19:07:03.8207145Z * [new branch] gh/tugsbayasgalan/89/base -> origin/gh/tugsbayasgalan/89/base 2022-03-17T19:07:03.8208278Z * [new branch] gh/tugsbayasgalan/89/head -> origin/gh/tugsbayasgalan/89/head 2022-03-17T19:07:03.8209687Z * [new branch] gh/tugsbayasgalan/89/orig -> origin/gh/tugsbayasgalan/89/orig 2022-03-17T19:07:03.8211686Z * [new branch] gh/tugsbayasgalan/9/base -> origin/gh/tugsbayasgalan/9/base 2022-03-17T19:07:03.8212895Z * [new branch] gh/tugsbayasgalan/9/head -> origin/gh/tugsbayasgalan/9/head 2022-03-17T19:07:03.8214593Z * [new branch] gh/tugsbayasgalan/9/orig -> origin/gh/tugsbayasgalan/9/orig 2022-03-17T19:07:03.8216558Z * [new branch] gh/tugsbayasgalan/90/base -> origin/gh/tugsbayasgalan/90/base 2022-03-17T19:07:03.8217775Z * [new branch] gh/tugsbayasgalan/90/head -> origin/gh/tugsbayasgalan/90/head 2022-03-17T19:07:03.8219479Z * [new branch] gh/tugsbayasgalan/90/orig -> origin/gh/tugsbayasgalan/90/orig 2022-03-17T19:07:03.8229609Z * [new branch] gh/tugsbayasgalan/91/base -> origin/gh/tugsbayasgalan/91/base 2022-03-17T19:07:03.8230947Z * [new branch] gh/tugsbayasgalan/91/head -> origin/gh/tugsbayasgalan/91/head 2022-03-17T19:07:03.8232644Z * [new branch] gh/tugsbayasgalan/91/orig -> origin/gh/tugsbayasgalan/91/orig 2022-03-17T19:07:03.8234776Z * [new branch] gh/varal7/24/base -> origin/gh/varal7/24/base 2022-03-17T19:07:03.8235936Z * [new branch] gh/varal7/24/orig -> origin/gh/varal7/24/orig 2022-03-17T19:07:03.8237757Z * [new branch] gh/varal7/25/base -> origin/gh/varal7/25/base 2022-03-17T19:07:03.8238941Z * [new branch] gh/varal7/25/orig -> origin/gh/varal7/25/orig 2022-03-17T19:07:03.8240716Z * [new branch] gh/varal7/26/base -> origin/gh/varal7/26/base 2022-03-17T19:07:03.8242080Z * [new branch] gh/varal7/26/orig -> origin/gh/varal7/26/orig 2022-03-17T19:07:03.8244178Z * [new branch] gh/varal7/31/base -> origin/gh/varal7/31/base 2022-03-17T19:07:03.8245326Z * [new branch] gh/varal7/31/orig -> origin/gh/varal7/31/orig 2022-03-17T19:07:03.8247602Z * [new branch] gh/vec4f/1/base -> origin/gh/vec4f/1/base 2022-03-17T19:07:03.8248847Z * [new branch] gh/vec4f/1/head -> origin/gh/vec4f/1/head 2022-03-17T19:07:03.8251131Z * [new branch] gh/vincentqb/16/base -> origin/gh/vincentqb/16/base 2022-03-17T19:07:03.8252384Z * [new branch] gh/vincentqb/16/head -> origin/gh/vincentqb/16/head 2022-03-17T19:07:03.8254460Z * [new branch] gh/vincentqb/21/base -> origin/gh/vincentqb/21/base 2022-03-17T19:07:03.8256392Z * [new branch] gh/vincentqb/21/orig -> origin/gh/vincentqb/21/orig 2022-03-17T19:07:03.8258185Z * [new branch] gh/vincentqb/24/base -> origin/gh/vincentqb/24/base 2022-03-17T19:07:03.8259381Z * [new branch] gh/vincentqb/24/orig -> origin/gh/vincentqb/24/orig 2022-03-17T19:07:03.8261875Z * [new branch] gh/vincentqb/30/base -> origin/gh/vincentqb/30/base 2022-03-17T19:07:03.8262889Z * [new branch] gh/vincentqb/30/orig -> origin/gh/vincentqb/30/orig 2022-03-17T19:07:03.8264885Z * [new branch] gh/vincentqb/31/base -> origin/gh/vincentqb/31/base 2022-03-17T19:07:03.8265927Z * [new branch] gh/vincentqb/31/head -> origin/gh/vincentqb/31/head 2022-03-17T19:07:03.8267598Z * [new branch] gh/vincentqb/31/orig -> origin/gh/vincentqb/31/orig 2022-03-17T19:07:03.8269657Z * [new branch] gh/vivekmig/1/base -> origin/gh/vivekmig/1/base 2022-03-17T19:07:03.8270791Z * [new branch] gh/vivekmig/1/head -> origin/gh/vivekmig/1/head 2022-03-17T19:07:03.8272396Z * [new branch] gh/vivekmig/1/orig -> origin/gh/vivekmig/1/orig 2022-03-17T19:07:03.8274514Z * [new branch] gh/vkuzo/10/orig -> origin/gh/vkuzo/10/orig 2022-03-17T19:07:03.8276326Z * [new branch] gh/vkuzo/15/base -> origin/gh/vkuzo/15/base 2022-03-17T19:07:03.8277989Z * [new branch] gh/vkuzo/15/head -> origin/gh/vkuzo/15/head 2022-03-17T19:07:03.8279258Z * [new branch] gh/vkuzo/15/orig -> origin/gh/vkuzo/15/orig 2022-03-17T19:07:03.8281386Z * [new branch] gh/vkuzo/23/base -> origin/gh/vkuzo/23/base 2022-03-17T19:07:03.8282556Z * [new branch] gh/vkuzo/23/orig -> origin/gh/vkuzo/23/orig 2022-03-17T19:07:03.8284627Z * [new branch] gh/vkuzo/28/base -> origin/gh/vkuzo/28/base 2022-03-17T19:07:03.8288334Z * [new branch] gh/vkuzo/28/orig -> origin/gh/vkuzo/28/orig 2022-03-17T19:07:03.8290110Z * [new branch] gh/vkuzo/3/base -> origin/gh/vkuzo/3/base 2022-03-17T19:07:03.8291306Z * [new branch] gh/vkuzo/3/orig -> origin/gh/vkuzo/3/orig 2022-03-17T19:07:03.8293709Z * [new branch] gh/vkuzo/349/base -> origin/gh/vkuzo/349/base 2022-03-17T19:07:03.8295353Z * [new branch] gh/vkuzo/349/head -> origin/gh/vkuzo/349/head 2022-03-17T19:07:03.8297082Z * [new branch] gh/vkuzo/349/orig -> origin/gh/vkuzo/349/orig 2022-03-17T19:07:03.8298823Z * [new branch] gh/vkuzo/393/base -> origin/gh/vkuzo/393/base 2022-03-17T19:07:03.8300620Z * [new branch] gh/vkuzo/393/head -> origin/gh/vkuzo/393/head 2022-03-17T19:07:03.8307202Z * [new branch] gh/vkuzo/393/orig -> origin/gh/vkuzo/393/orig 2022-03-17T19:07:03.8308926Z * [new branch] gh/vkuzo/4/base -> origin/gh/vkuzo/4/base 2022-03-17T19:07:03.8310284Z * [new branch] gh/vkuzo/4/orig -> origin/gh/vkuzo/4/orig 2022-03-17T19:07:03.8312245Z * [new branch] gh/vkuzo/439/base -> origin/gh/vkuzo/439/base 2022-03-17T19:07:03.8315892Z * [new branch] gh/vkuzo/439/head -> origin/gh/vkuzo/439/head 2022-03-17T19:07:03.8319151Z * [new branch] gh/vkuzo/439/orig -> origin/gh/vkuzo/439/orig 2022-03-17T19:07:03.8321948Z * [new branch] gh/vkuzo/474/base -> origin/gh/vkuzo/474/base 2022-03-17T19:07:03.8323687Z * [new branch] gh/vkuzo/474/head -> origin/gh/vkuzo/474/head 2022-03-17T19:07:03.8324827Z * [new branch] gh/vkuzo/474/orig -> origin/gh/vkuzo/474/orig 2022-03-17T19:07:03.8327043Z * [new branch] gh/vkuzo/56/base -> origin/gh/vkuzo/56/base 2022-03-17T19:07:03.8328643Z * [new branch] gh/vkuzo/56/head -> origin/gh/vkuzo/56/head 2022-03-17T19:07:03.8330017Z * [new branch] gh/vkuzo/56/orig -> origin/gh/vkuzo/56/orig 2022-03-17T19:07:03.8332308Z * [new branch] gh/vkuzo/8/base -> origin/gh/vkuzo/8/base 2022-03-17T19:07:03.8333408Z * [new branch] gh/vkuzo/8/orig -> origin/gh/vkuzo/8/orig 2022-03-17T19:07:03.8335360Z * [new branch] gh/vkuzo/9/orig -> origin/gh/vkuzo/9/orig 2022-03-17T19:07:03.8337572Z * [new branch] gh/wanchaol/126/base -> origin/gh/wanchaol/126/base 2022-03-17T19:07:03.8339183Z * [new branch] gh/wanchaol/126/head -> origin/gh/wanchaol/126/head 2022-03-17T19:07:03.8340494Z * [new branch] gh/wanchaol/126/orig -> origin/gh/wanchaol/126/orig 2022-03-17T19:07:03.8344069Z * [new branch] gh/wanchaol/134/base -> origin/gh/wanchaol/134/base 2022-03-17T19:07:03.8346514Z * [new branch] gh/wanchaol/134/head -> origin/gh/wanchaol/134/head 2022-03-17T19:07:03.8347895Z * [new branch] gh/wanchaol/134/orig -> origin/gh/wanchaol/134/orig 2022-03-17T19:07:03.8349899Z * [new branch] gh/wanchaol/149/base -> origin/gh/wanchaol/149/base 2022-03-17T19:07:03.8351045Z * [new branch] gh/wanchaol/149/head -> origin/gh/wanchaol/149/head 2022-03-17T19:07:03.8352780Z * [new branch] gh/wanchaol/149/orig -> origin/gh/wanchaol/149/orig 2022-03-17T19:07:03.8354608Z * [new branch] gh/wanchaol/150/base -> origin/gh/wanchaol/150/base 2022-03-17T19:07:03.8356667Z * [new branch] gh/wanchaol/150/head -> origin/gh/wanchaol/150/head 2022-03-17T19:07:03.8358257Z * [new branch] gh/wanchaol/150/orig -> origin/gh/wanchaol/150/orig 2022-03-17T19:07:03.8360220Z * [new branch] gh/wanchaol/169/base -> origin/gh/wanchaol/169/base 2022-03-17T19:07:03.8361984Z * [new branch] gh/wanchaol/169/head -> origin/gh/wanchaol/169/head 2022-03-17T19:07:03.8363742Z * [new branch] gh/wanchaol/189/base -> origin/gh/wanchaol/189/base 2022-03-17T19:07:03.8365171Z * [new branch] gh/wanchaol/189/head -> origin/gh/wanchaol/189/head 2022-03-17T19:07:03.8366903Z * [new branch] gh/wanchaol/189/orig -> origin/gh/wanchaol/189/orig 2022-03-17T19:07:03.8368891Z * [new branch] gh/wanchaol/200/base -> origin/gh/wanchaol/200/base 2022-03-17T19:07:03.8370260Z * [new branch] gh/wanchaol/200/head -> origin/gh/wanchaol/200/head 2022-03-17T19:07:03.8371628Z * [new branch] gh/wanchaol/200/orig -> origin/gh/wanchaol/200/orig 2022-03-17T19:07:03.8373499Z * [new branch] gh/wanchaol/203/base -> origin/gh/wanchaol/203/base 2022-03-17T19:07:03.8375089Z * [new branch] gh/wanchaol/203/head -> origin/gh/wanchaol/203/head 2022-03-17T19:07:03.8377131Z * [new branch] gh/wanchaol/204/base -> origin/gh/wanchaol/204/base 2022-03-17T19:07:03.8378286Z * [new branch] gh/wanchaol/204/head -> origin/gh/wanchaol/204/head 2022-03-17T19:07:03.8379961Z * [new branch] gh/wanchaol/204/orig -> origin/gh/wanchaol/204/orig 2022-03-17T19:07:03.8382475Z * [new branch] gh/wanchaol/206/base -> origin/gh/wanchaol/206/base 2022-03-17T19:07:03.8383745Z * [new branch] gh/wanchaol/206/head -> origin/gh/wanchaol/206/head 2022-03-17T19:07:03.8385526Z * [new branch] gh/wanchaol/206/orig -> origin/gh/wanchaol/206/orig 2022-03-17T19:07:03.8387008Z * [new branch] gh/wanchaol/207/base -> origin/gh/wanchaol/207/base 2022-03-17T19:07:03.8388952Z * [new branch] gh/wanchaol/207/head -> origin/gh/wanchaol/207/head 2022-03-17T19:07:03.8390146Z * [new branch] gh/wanchaol/207/orig -> origin/gh/wanchaol/207/orig 2022-03-17T19:07:03.8392910Z * [new branch] gh/wanchaol/44/base -> origin/gh/wanchaol/44/base 2022-03-17T19:07:03.8394065Z * [new branch] gh/wanchaol/44/head -> origin/gh/wanchaol/44/head 2022-03-17T19:07:03.8396650Z * [new branch] gh/wanchaol/69/base -> origin/gh/wanchaol/69/base 2022-03-17T19:07:03.8397844Z * [new branch] gh/wanchaol/69/head -> origin/gh/wanchaol/69/head 2022-03-17T19:07:03.8399605Z * [new branch] gh/wanchaol/69/orig -> origin/gh/wanchaol/69/orig 2022-03-17T19:07:03.8401520Z * [new branch] gh/wanchaol/75/base -> origin/gh/wanchaol/75/base 2022-03-17T19:07:03.8402603Z * [new branch] gh/wanchaol/75/head -> origin/gh/wanchaol/75/head 2022-03-17T19:07:03.8403980Z * [new branch] gh/wanchaol/75/orig -> origin/gh/wanchaol/75/orig 2022-03-17T19:07:03.8406324Z * [new branch] gh/wenlei.xie@gmail.com/1/base -> origin/gh/wenlei.xie@gmail.com/1/base 2022-03-17T19:07:03.8407539Z * [new branch] gh/wenlei.xie@gmail.com/1/orig -> origin/gh/wenlei.xie@gmail.com/1/orig 2022-03-17T19:07:03.8409551Z * [new branch] gh/wenlei.xie@gmail.com/10/base -> origin/gh/wenlei.xie@gmail.com/10/base 2022-03-17T19:07:03.8410814Z * [new branch] gh/wenlei.xie@gmail.com/10/orig -> origin/gh/wenlei.xie@gmail.com/10/orig 2022-03-17T19:07:03.8412793Z * [new branch] gh/wenlei.xie@gmail.com/11/base -> origin/gh/wenlei.xie@gmail.com/11/base 2022-03-17T19:07:03.8413916Z * [new branch] gh/wenlei.xie@gmail.com/11/orig -> origin/gh/wenlei.xie@gmail.com/11/orig 2022-03-17T19:07:03.8415968Z * [new branch] gh/wenlei.xie@gmail.com/12/base -> origin/gh/wenlei.xie@gmail.com/12/base 2022-03-17T19:07:03.8417052Z * [new branch] gh/wenlei.xie@gmail.com/12/orig -> origin/gh/wenlei.xie@gmail.com/12/orig 2022-03-17T19:07:03.8419203Z * [new branch] gh/wenlei.xie@gmail.com/13/base -> origin/gh/wenlei.xie@gmail.com/13/base 2022-03-17T19:07:03.8420698Z * [new branch] gh/wenlei.xie@gmail.com/13/orig -> origin/gh/wenlei.xie@gmail.com/13/orig 2022-03-17T19:07:03.8423122Z * [new branch] gh/wenlei.xie@gmail.com/14/base -> origin/gh/wenlei.xie@gmail.com/14/base 2022-03-17T19:07:03.8424346Z * [new branch] gh/wenlei.xie@gmail.com/14/orig -> origin/gh/wenlei.xie@gmail.com/14/orig 2022-03-17T19:07:03.8426663Z * [new branch] gh/wenlei.xie@gmail.com/15/base -> origin/gh/wenlei.xie@gmail.com/15/base 2022-03-17T19:07:03.8428064Z * [new branch] gh/wenlei.xie@gmail.com/15/orig -> origin/gh/wenlei.xie@gmail.com/15/orig 2022-03-17T19:07:03.8430217Z * [new branch] gh/wenlei.xie@gmail.com/16/base -> origin/gh/wenlei.xie@gmail.com/16/base 2022-03-17T19:07:03.8431393Z * [new branch] gh/wenlei.xie@gmail.com/16/orig -> origin/gh/wenlei.xie@gmail.com/16/orig 2022-03-17T19:07:03.8433493Z * [new branch] gh/wenlei.xie@gmail.com/17/base -> origin/gh/wenlei.xie@gmail.com/17/base 2022-03-17T19:07:03.8434659Z * [new branch] gh/wenlei.xie@gmail.com/17/orig -> origin/gh/wenlei.xie@gmail.com/17/orig 2022-03-17T19:07:03.8436623Z * [new branch] gh/wenlei.xie@gmail.com/18/base -> origin/gh/wenlei.xie@gmail.com/18/base 2022-03-17T19:07:03.8437812Z * [new branch] gh/wenlei.xie@gmail.com/18/orig -> origin/gh/wenlei.xie@gmail.com/18/orig 2022-03-17T19:07:03.8439869Z * [new branch] gh/wenlei.xie@gmail.com/2/base -> origin/gh/wenlei.xie@gmail.com/2/base 2022-03-17T19:07:03.8440986Z * [new branch] gh/wenlei.xie@gmail.com/2/orig -> origin/gh/wenlei.xie@gmail.com/2/orig 2022-03-17T19:07:03.8442445Z * [new branch] gh/wenlei.xie@gmail.com/3/base -> origin/gh/wenlei.xie@gmail.com/3/base 2022-03-17T19:07:03.8443801Z * [new branch] gh/wenlei.xie@gmail.com/3/orig -> origin/gh/wenlei.xie@gmail.com/3/orig 2022-03-17T19:07:03.8446036Z * [new branch] gh/wenlei.xie@gmail.com/4/base -> origin/gh/wenlei.xie@gmail.com/4/base 2022-03-17T19:07:03.8447198Z * [new branch] gh/wenlei.xie@gmail.com/4/orig -> origin/gh/wenlei.xie@gmail.com/4/orig 2022-03-17T19:07:03.8449243Z * [new branch] gh/wenlei.xie@gmail.com/5/base -> origin/gh/wenlei.xie@gmail.com/5/base 2022-03-17T19:07:03.8450379Z * [new branch] gh/wenlei.xie@gmail.com/5/orig -> origin/gh/wenlei.xie@gmail.com/5/orig 2022-03-17T19:07:03.8452703Z * [new branch] gh/wenlei.xie@gmail.com/6/base -> origin/gh/wenlei.xie@gmail.com/6/base 2022-03-17T19:07:03.8453612Z * [new branch] gh/wenlei.xie@gmail.com/6/orig -> origin/gh/wenlei.xie@gmail.com/6/orig 2022-03-17T19:07:03.8455331Z * [new branch] gh/wenlei.xie@gmail.com/7/base -> origin/gh/wenlei.xie@gmail.com/7/base 2022-03-17T19:07:03.8456535Z * [new branch] gh/wenlei.xie@gmail.com/7/head -> origin/gh/wenlei.xie@gmail.com/7/head 2022-03-17T19:07:03.8457977Z * [new branch] gh/wenlei.xie@gmail.com/7/orig -> origin/gh/wenlei.xie@gmail.com/7/orig 2022-03-17T19:07:03.8459849Z * [new branch] gh/wenlei.xie@gmail.com/8/base -> origin/gh/wenlei.xie@gmail.com/8/base 2022-03-17T19:07:03.8461204Z * [new branch] gh/wenlei.xie@gmail.com/8/head -> origin/gh/wenlei.xie@gmail.com/8/head 2022-03-17T19:07:03.8463235Z * [new branch] gh/wenlei.xie@gmail.com/8/orig -> origin/gh/wenlei.xie@gmail.com/8/orig 2022-03-17T19:07:03.8464617Z * [new branch] gh/wenlei.xie@gmail.com/9/base -> origin/gh/wenlei.xie@gmail.com/9/base 2022-03-17T19:07:03.8466048Z * [new branch] gh/wenlei.xie@gmail.com/9/head -> origin/gh/wenlei.xie@gmail.com/9/head 2022-03-17T19:07:03.8467704Z * [new branch] gh/wenlei.xie@gmail.com/9/orig -> origin/gh/wenlei.xie@gmail.com/9/orig 2022-03-17T19:07:03.8469868Z * [new branch] gh/wenleix/1/base -> origin/gh/wenleix/1/base 2022-03-17T19:07:03.8471174Z * [new branch] gh/wenleix/1/orig -> origin/gh/wenleix/1/orig 2022-03-17T19:07:03.8473505Z * [new branch] gh/wenleix/18/base -> origin/gh/wenleix/18/base 2022-03-17T19:07:03.8474914Z * [new branch] gh/wenleix/18/head -> origin/gh/wenleix/18/head 2022-03-17T19:07:03.8477303Z * [new branch] gh/wenleix/18/orig -> origin/gh/wenleix/18/orig 2022-03-17T19:07:03.8478886Z * [new branch] gh/wenleix/20/base -> origin/gh/wenleix/20/base 2022-03-17T19:07:03.8481049Z * [new branch] gh/wenleix/20/head -> origin/gh/wenleix/20/head 2022-03-17T19:07:03.8482195Z * [new branch] gh/wenleix/20/orig -> origin/gh/wenleix/20/orig 2022-03-17T19:07:03.8484159Z * [new branch] gh/wenleix/21/base -> origin/gh/wenleix/21/base 2022-03-17T19:07:03.8485942Z * [new branch] gh/wenleix/21/head -> origin/gh/wenleix/21/head 2022-03-17T19:07:03.8487079Z * [new branch] gh/wenleix/21/orig -> origin/gh/wenleix/21/orig 2022-03-17T19:07:03.8489096Z * [new branch] gh/wenleix/23/base -> origin/gh/wenleix/23/base 2022-03-17T19:07:03.8490208Z * [new branch] gh/wenleix/23/head -> origin/gh/wenleix/23/head 2022-03-17T19:07:03.8492043Z * [new branch] gh/wenleix/23/orig -> origin/gh/wenleix/23/orig 2022-03-17T19:07:03.8494040Z * [new branch] gh/wenleix/24/base -> origin/gh/wenleix/24/base 2022-03-17T19:07:03.8495312Z * [new branch] gh/wenleix/24/head -> origin/gh/wenleix/24/head 2022-03-17T19:07:03.8497341Z * [new branch] gh/wenleix/24/orig -> origin/gh/wenleix/24/orig 2022-03-17T19:07:03.8498652Z * [new branch] gh/wenleix/3/base -> origin/gh/wenleix/3/base 2022-03-17T19:07:03.8499916Z * [new branch] gh/wenleix/3/orig -> origin/gh/wenleix/3/orig 2022-03-17T19:07:03.8502692Z * [new branch] gh/wenleix/30/base -> origin/gh/wenleix/30/base 2022-03-17T19:07:03.8503750Z * [new branch] gh/wenleix/30/head -> origin/gh/wenleix/30/head 2022-03-17T19:07:03.8505597Z * [new branch] gh/wenleix/30/orig -> origin/gh/wenleix/30/orig 2022-03-17T19:07:03.8507908Z * [new branch] gh/wenleix/31/base -> origin/gh/wenleix/31/base 2022-03-17T19:07:03.8509210Z * [new branch] gh/wenleix/31/head -> origin/gh/wenleix/31/head 2022-03-17T19:07:03.8510599Z * [new branch] gh/wenleix/31/orig -> origin/gh/wenleix/31/orig 2022-03-17T19:07:03.8512811Z * [new branch] gh/wenleix/32/base -> origin/gh/wenleix/32/base 2022-03-17T19:07:03.8514013Z * [new branch] gh/wenleix/32/head -> origin/gh/wenleix/32/head 2022-03-17T19:07:03.8515528Z * [new branch] gh/wenleix/32/orig -> origin/gh/wenleix/32/orig 2022-03-17T19:07:03.8517432Z * [new branch] gh/wenleix/8/base -> origin/gh/wenleix/8/base 2022-03-17T19:07:03.8518598Z * [new branch] gh/wenleix/8/orig -> origin/gh/wenleix/8/orig 2022-03-17T19:07:03.8520968Z * [new branch] gh/xcheng16/1/base -> origin/gh/xcheng16/1/base 2022-03-17T19:07:03.8522108Z * [new branch] gh/xcheng16/1/head -> origin/gh/xcheng16/1/head 2022-03-17T19:07:03.8524029Z * [new branch] gh/xcheng16/16/base -> origin/gh/xcheng16/16/base 2022-03-17T19:07:03.8525348Z * [new branch] gh/xcheng16/16/head -> origin/gh/xcheng16/16/head 2022-03-17T19:07:03.8527048Z * [new branch] gh/xcheng16/16/orig -> origin/gh/xcheng16/16/orig 2022-03-17T19:07:03.8529147Z * [new branch] gh/xcheng16/19/base -> origin/gh/xcheng16/19/base 2022-03-17T19:07:03.8530379Z * [new branch] gh/xcheng16/19/head -> origin/gh/xcheng16/19/head 2022-03-17T19:07:03.8531859Z * [new branch] gh/xcheng16/19/orig -> origin/gh/xcheng16/19/orig 2022-03-17T19:07:03.8533795Z * [new branch] gh/xcheng16/23/base -> origin/gh/xcheng16/23/base 2022-03-17T19:07:03.8534975Z * [new branch] gh/xcheng16/23/head -> origin/gh/xcheng16/23/head 2022-03-17T19:07:03.8536725Z * [new branch] gh/xcheng16/24/base -> origin/gh/xcheng16/24/base 2022-03-17T19:07:03.8537804Z * [new branch] gh/xcheng16/24/head -> origin/gh/xcheng16/24/head 2022-03-17T19:07:03.8539447Z * [new branch] gh/xcheng16/25/base -> origin/gh/xcheng16/25/base 2022-03-17T19:07:03.8540578Z * [new branch] gh/xcheng16/25/head -> origin/gh/xcheng16/25/head 2022-03-17T19:07:03.8542712Z * [new branch] gh/xcheng16/26/base -> origin/gh/xcheng16/26/base 2022-03-17T19:07:03.8543896Z * [new branch] gh/xcheng16/26/head -> origin/gh/xcheng16/26/head 2022-03-17T19:07:03.8545986Z * [new branch] gh/xcheng16/27/base -> origin/gh/xcheng16/27/base 2022-03-17T19:07:03.8547167Z * [new branch] gh/xcheng16/27/head -> origin/gh/xcheng16/27/head 2022-03-17T19:07:03.8548794Z * [new branch] gh/xcheng16/28/base -> origin/gh/xcheng16/28/base 2022-03-17T19:07:03.8550077Z * [new branch] gh/xcheng16/28/head -> origin/gh/xcheng16/28/head 2022-03-17T19:07:03.8551860Z * [new branch] gh/xcheng16/29/base -> origin/gh/xcheng16/29/base 2022-03-17T19:07:03.8552885Z * [new branch] gh/xcheng16/29/head -> origin/gh/xcheng16/29/head 2022-03-17T19:07:03.8554604Z * [new branch] gh/xcheng16/30/base -> origin/gh/xcheng16/30/base 2022-03-17T19:07:03.8555676Z * [new branch] gh/xcheng16/30/head -> origin/gh/xcheng16/30/head 2022-03-17T19:07:03.8557807Z * [new branch] gh/xcheng16/5/base -> origin/gh/xcheng16/5/base 2022-03-17T19:07:03.8558958Z * [new branch] gh/xcheng16/5/head -> origin/gh/xcheng16/5/head 2022-03-17T19:07:03.8561129Z * [new branch] gh/xcheng16/7/head -> origin/gh/xcheng16/7/head 2022-03-17T19:07:03.8563802Z * [new branch] gh/xiaobingsuper/1/base -> origin/gh/xiaobingsuper/1/base 2022-03-17T19:07:03.8564944Z * [new branch] gh/xiaobingsuper/1/orig -> origin/gh/xiaobingsuper/1/orig 2022-03-17T19:07:03.8566960Z * [new branch] gh/xiaobingsuper/2/base -> origin/gh/xiaobingsuper/2/base 2022-03-17T19:07:03.8568022Z * [new branch] gh/xiaobingsuper/2/head -> origin/gh/xiaobingsuper/2/head 2022-03-17T19:07:03.8570174Z * [new branch] gh/xiaobingsuper/6/base -> origin/gh/xiaobingsuper/6/base 2022-03-17T19:07:03.8571318Z * [new branch] gh/xiaobingsuper/6/head -> origin/gh/xiaobingsuper/6/head 2022-03-17T19:07:03.8573075Z * [new branch] gh/xiaobingsuper/6/orig -> origin/gh/xiaobingsuper/6/orig 2022-03-17T19:07:03.8574581Z * [new branch] gh/xiaobingsuper/8/base -> origin/gh/xiaobingsuper/8/base 2022-03-17T19:07:03.8575928Z * [new branch] gh/xiaobingsuper/8/orig -> origin/gh/xiaobingsuper/8/orig 2022-03-17T19:07:03.8578411Z * [new branch] gh/xta0/114/base -> origin/gh/xta0/114/base 2022-03-17T19:07:03.8579572Z * [new branch] gh/xta0/114/head -> origin/gh/xta0/114/head 2022-03-17T19:07:03.8581022Z * [new branch] gh/xta0/114/orig -> origin/gh/xta0/114/orig 2022-03-17T19:07:03.8583454Z * [new branch] gh/xta0/117/base -> origin/gh/xta0/117/base 2022-03-17T19:07:03.8584668Z * [new branch] gh/xta0/117/head -> origin/gh/xta0/117/head 2022-03-17T19:07:03.8586274Z * [new branch] gh/xta0/118/base -> origin/gh/xta0/118/base 2022-03-17T19:07:03.8587487Z * [new branch] gh/xta0/118/head -> origin/gh/xta0/118/head 2022-03-17T19:07:03.8589914Z * [new branch] gh/xta0/129/base -> origin/gh/xta0/129/base 2022-03-17T19:07:03.8591185Z * [new branch] gh/xta0/129/head -> origin/gh/xta0/129/head 2022-03-17T19:07:03.8592797Z * [new branch] gh/xta0/129/orig -> origin/gh/xta0/129/orig 2022-03-17T19:07:03.8594639Z * [new branch] gh/xta0/138/base -> origin/gh/xta0/138/base 2022-03-17T19:07:03.8595787Z * [new branch] gh/xta0/138/head -> origin/gh/xta0/138/head 2022-03-17T19:07:03.8598042Z * [new branch] gh/xta0/142/base -> origin/gh/xta0/142/base 2022-03-17T19:07:03.8599164Z * [new branch] gh/xta0/142/head -> origin/gh/xta0/142/head 2022-03-17T19:07:03.8600530Z * [new branch] gh/xta0/142/orig -> origin/gh/xta0/142/orig 2022-03-17T19:07:03.8602535Z * [new branch] gh/xta0/144/base -> origin/gh/xta0/144/base 2022-03-17T19:07:03.8603601Z * [new branch] gh/xta0/144/head -> origin/gh/xta0/144/head 2022-03-17T19:07:03.8605279Z * [new branch] gh/xta0/144/orig -> origin/gh/xta0/144/orig 2022-03-17T19:07:03.8606885Z * [new branch] gh/xta0/145/base -> origin/gh/xta0/145/base 2022-03-17T19:07:03.8608142Z * [new branch] gh/xta0/145/head -> origin/gh/xta0/145/head 2022-03-17T19:07:03.8609729Z * [new branch] gh/xta0/145/orig -> origin/gh/xta0/145/orig 2022-03-17T19:07:03.8611595Z * [new branch] gh/xta0/2/base -> origin/gh/xta0/2/base 2022-03-17T19:07:03.8613256Z * [new branch] gh/xta0/2/head -> origin/gh/xta0/2/head 2022-03-17T19:07:03.8615034Z * [new branch] gh/xta0/2/orig -> origin/gh/xta0/2/orig 2022-03-17T19:07:03.8616897Z * [new branch] gh/xta0/29/base -> origin/gh/xta0/29/base 2022-03-17T19:07:03.8618686Z * [new branch] gh/xta0/29/head -> origin/gh/xta0/29/head 2022-03-17T19:07:03.8619862Z * [new branch] gh/xta0/29/orig -> origin/gh/xta0/29/orig 2022-03-17T19:07:03.8622569Z * [new branch] gh/xta0/3/base -> origin/gh/xta0/3/base 2022-03-17T19:07:03.8623937Z * [new branch] gh/xta0/3/head -> origin/gh/xta0/3/head 2022-03-17T19:07:03.8625579Z * [new branch] gh/xta0/3/orig -> origin/gh/xta0/3/orig 2022-03-17T19:07:03.8626975Z * [new branch] gh/xta0/4/base -> origin/gh/xta0/4/base 2022-03-17T19:07:03.8628530Z * [new branch] gh/xta0/4/head -> origin/gh/xta0/4/head 2022-03-17T19:07:03.8629864Z * [new branch] gh/xta0/4/orig -> origin/gh/xta0/4/orig 2022-03-17T19:07:03.8631836Z * [new branch] gh/xta0/47/base -> origin/gh/xta0/47/base 2022-03-17T19:07:03.8633424Z * [new branch] gh/xta0/47/head -> origin/gh/xta0/47/head 2022-03-17T19:07:03.8634502Z * [new branch] gh/xta0/47/orig -> origin/gh/xta0/47/orig 2022-03-17T19:07:03.8636498Z * [new branch] gh/xta0/48/base -> origin/gh/xta0/48/base 2022-03-17T19:07:03.8637652Z * [new branch] gh/xta0/48/head -> origin/gh/xta0/48/head 2022-03-17T19:07:03.8639585Z * [new branch] gh/xta0/48/orig -> origin/gh/xta0/48/orig 2022-03-17T19:07:03.8641273Z * [new branch] gh/xta0/61/base -> origin/gh/xta0/61/base 2022-03-17T19:07:03.8642472Z * [new branch] gh/xta0/61/head -> origin/gh/xta0/61/head 2022-03-17T19:07:03.8644510Z * [new branch] gh/xta0/72/base -> origin/gh/xta0/72/base 2022-03-17T19:07:03.8646254Z * [new branch] gh/xta0/72/head -> origin/gh/xta0/72/head 2022-03-17T19:07:03.8647050Z * [new branch] gh/xta0/72/orig -> origin/gh/xta0/72/orig 2022-03-17T19:07:03.8649271Z * [new branch] gh/xta0/89/base -> origin/gh/xta0/89/base 2022-03-17T19:07:03.8651220Z * [new branch] gh/xta0/89/head -> origin/gh/xta0/89/head 2022-03-17T19:07:03.8653261Z * [new branch] gh/xuhdev/10/base -> origin/gh/xuhdev/10/base 2022-03-17T19:07:03.8655492Z * [new branch] gh/xuhdev/14/base -> origin/gh/xuhdev/14/base 2022-03-17T19:07:03.8656711Z * [new branch] gh/xuhdev/14/head -> origin/gh/xuhdev/14/head 2022-03-17T19:07:03.8658478Z * [new branch] gh/xuhdev/14/orig -> origin/gh/xuhdev/14/orig 2022-03-17T19:07:03.8660193Z * [new branch] gh/xuhdev/15/base -> origin/gh/xuhdev/15/base 2022-03-17T19:07:03.8662436Z * [new branch] gh/xuhdev/26/base -> origin/gh/xuhdev/26/base 2022-03-17T19:07:03.8663607Z * [new branch] gh/xuhdev/26/orig -> origin/gh/xuhdev/26/orig 2022-03-17T19:07:03.8665630Z * [new branch] gh/xuhdev/28/base -> origin/gh/xuhdev/28/base 2022-03-17T19:07:03.8667171Z * [new branch] gh/xuhdev/28/head -> origin/gh/xuhdev/28/head 2022-03-17T19:07:03.8668401Z * [new branch] gh/xuhdev/28/orig -> origin/gh/xuhdev/28/orig 2022-03-17T19:07:03.8670505Z * [new branch] gh/xuhdev/32/base -> origin/gh/xuhdev/32/base 2022-03-17T19:07:03.8671667Z * [new branch] gh/xuhdev/32/orig -> origin/gh/xuhdev/32/orig 2022-03-17T19:07:03.8673609Z * [new branch] gh/xuhdev/33/base -> origin/gh/xuhdev/33/base 2022-03-17T19:07:03.8674854Z * [new branch] gh/xuhdev/33/orig -> origin/gh/xuhdev/33/orig 2022-03-17T19:07:03.8677046Z * [new branch] gh/xuhdev/34/base -> origin/gh/xuhdev/34/base 2022-03-17T19:07:03.8678382Z * [new branch] gh/xuhdev/34/orig -> origin/gh/xuhdev/34/orig 2022-03-17T19:07:03.8680229Z * [new branch] gh/xuhdev/36/base -> origin/gh/xuhdev/36/base 2022-03-17T19:07:03.8681580Z * [new branch] gh/xuhdev/36/orig -> origin/gh/xuhdev/36/orig 2022-03-17T19:07:03.8683569Z * [new branch] gh/xuhdev/38/base -> origin/gh/xuhdev/38/base 2022-03-17T19:07:03.8684813Z * [new branch] gh/xuhdev/38/orig -> origin/gh/xuhdev/38/orig 2022-03-17T19:07:03.8686743Z * [new branch] gh/xuhdev/4/base -> origin/gh/xuhdev/4/base 2022-03-17T19:07:03.8688718Z * [new branch] gh/xuhdev/40/base -> origin/gh/xuhdev/40/base 2022-03-17T19:07:03.8690802Z * [new branch] gh/xuhdev/40/orig -> origin/gh/xuhdev/40/orig 2022-03-17T19:07:03.8693222Z * [new branch] gh/xuhdev/41/orig -> origin/gh/xuhdev/41/orig 2022-03-17T19:07:03.8695230Z * [new branch] gh/xuhdev/42/base -> origin/gh/xuhdev/42/base 2022-03-17T19:07:03.8696919Z * [new branch] gh/xuhdev/42/orig -> origin/gh/xuhdev/42/orig 2022-03-17T19:07:03.8698732Z * [new branch] gh/xuhdev/50/base -> origin/gh/xuhdev/50/base 2022-03-17T19:07:03.8699902Z * [new branch] gh/xuhdev/50/head -> origin/gh/xuhdev/50/head 2022-03-17T19:07:03.8702513Z * [new branch] gh/xuhdev/50/orig -> origin/gh/xuhdev/50/orig 2022-03-17T19:07:03.8704467Z * [new branch] gh/xuhdev/52/base -> origin/gh/xuhdev/52/base 2022-03-17T19:07:03.8705605Z * [new branch] gh/xuhdev/52/head -> origin/gh/xuhdev/52/head 2022-03-17T19:07:03.8707247Z * [new branch] gh/xuhdev/52/orig -> origin/gh/xuhdev/52/orig 2022-03-17T19:07:03.8708988Z * [new branch] gh/xuhdev/53/base -> origin/gh/xuhdev/53/base 2022-03-17T19:07:03.8710844Z * [new branch] gh/xuhdev/55/base -> origin/gh/xuhdev/55/base 2022-03-17T19:07:03.8712587Z * [new branch] gh/xuhdev/55/orig -> origin/gh/xuhdev/55/orig 2022-03-17T19:07:03.8714204Z * [new branch] gh/xuhdev/56/base -> origin/gh/xuhdev/56/base 2022-03-17T19:07:03.8715391Z * [new branch] gh/xuhdev/56/orig -> origin/gh/xuhdev/56/orig 2022-03-17T19:07:03.8717545Z * [new branch] gh/xuhdev/58/base -> origin/gh/xuhdev/58/base 2022-03-17T19:07:03.8718681Z * [new branch] gh/xuhdev/58/orig -> origin/gh/xuhdev/58/orig 2022-03-17T19:07:03.8720651Z * [new branch] gh/xuhdev/59/orig -> origin/gh/xuhdev/59/orig 2022-03-17T19:07:03.8722477Z * [new branch] gh/xuhdev/60/orig -> origin/gh/xuhdev/60/orig 2022-03-17T19:07:03.8724075Z * [new branch] gh/xuhdev/61/base -> origin/gh/xuhdev/61/base 2022-03-17T19:07:03.8725318Z * [new branch] gh/xuhdev/61/orig -> origin/gh/xuhdev/61/orig 2022-03-17T19:07:03.8727170Z * [new branch] gh/xuhdev/67/orig -> origin/gh/xuhdev/67/orig 2022-03-17T19:07:03.8728891Z * [new branch] gh/xuhdev/69/base -> origin/gh/xuhdev/69/base 2022-03-17T19:07:03.8730051Z * [new branch] gh/xuhdev/69/orig -> origin/gh/xuhdev/69/orig 2022-03-17T19:07:03.8731993Z * [new branch] gh/xuhdev/70/orig -> origin/gh/xuhdev/70/orig 2022-03-17T19:07:03.8734070Z * [new branch] gh/xuhdev/73/base -> origin/gh/xuhdev/73/base 2022-03-17T19:07:03.8735123Z * [new branch] gh/xuhdev/73/orig -> origin/gh/xuhdev/73/orig 2022-03-17T19:07:03.8737716Z * [new branch] gh/xuhdev/74/orig -> origin/gh/xuhdev/74/orig 2022-03-17T19:07:03.8739462Z * [new branch] gh/xuhdev/77/base -> origin/gh/xuhdev/77/base 2022-03-17T19:07:03.8740900Z * [new branch] gh/xuhdev/77/orig -> origin/gh/xuhdev/77/orig 2022-03-17T19:07:03.8743321Z * [new branch] gh/xuhdev/78/base -> origin/gh/xuhdev/78/base 2022-03-17T19:07:03.8744966Z * [new branch] gh/xuhdev/78/orig -> origin/gh/xuhdev/78/orig 2022-03-17T19:07:03.8746808Z * [new branch] gh/xuhdev/79/base -> origin/gh/xuhdev/79/base 2022-03-17T19:07:03.8747980Z * [new branch] gh/xuhdev/79/orig -> origin/gh/xuhdev/79/orig 2022-03-17T19:07:03.8750025Z * [new branch] gh/xuhdev/81/base -> origin/gh/xuhdev/81/base 2022-03-17T19:07:03.8751186Z * [new branch] gh/xuhdev/81/orig -> origin/gh/xuhdev/81/orig 2022-03-17T19:07:03.8753314Z * [new branch] gh/xuhdev/82/base -> origin/gh/xuhdev/82/base 2022-03-17T19:07:03.8755455Z * [new branch] gh/xuhdev/82/orig -> origin/gh/xuhdev/82/orig 2022-03-17T19:07:03.8758325Z * [new branch] gh/xuhdev/83/base -> origin/gh/xuhdev/83/base 2022-03-17T19:07:03.8759554Z * [new branch] gh/xuhdev/83/orig -> origin/gh/xuhdev/83/orig 2022-03-17T19:07:03.8761535Z * [new branch] gh/xuhdev/87/base -> origin/gh/xuhdev/87/base 2022-03-17T19:07:03.8762671Z * [new branch] gh/xuhdev/87/orig -> origin/gh/xuhdev/87/orig 2022-03-17T19:07:03.8764636Z * [new branch] gh/xuhdev/88/base -> origin/gh/xuhdev/88/base 2022-03-17T19:07:03.8765897Z * [new branch] gh/xuhdev/88/orig -> origin/gh/xuhdev/88/orig 2022-03-17T19:07:03.8767861Z * [new branch] gh/xuhdev/89/base -> origin/gh/xuhdev/89/base 2022-03-17T19:07:03.8768989Z * [new branch] gh/xuhdev/89/orig -> origin/gh/xuhdev/89/orig 2022-03-17T19:07:03.8771606Z * [new branch] gh/xuhdev/9/base -> origin/gh/xuhdev/9/base 2022-03-17T19:07:03.8773235Z * [new branch] gh/xuhdev/9/head -> origin/gh/xuhdev/9/head 2022-03-17T19:07:03.8774401Z * [new branch] gh/xuhdev/9/orig -> origin/gh/xuhdev/9/orig 2022-03-17T19:07:03.8776625Z * [new branch] gh/xuhdev/96/base -> origin/gh/xuhdev/96/base 2022-03-17T19:07:03.8778224Z * [new branch] gh/xuhdev/96/head -> origin/gh/xuhdev/96/head 2022-03-17T19:07:03.8780167Z * [new branch] gh/xuhdev/96/orig -> origin/gh/xuhdev/96/orig 2022-03-17T19:07:03.8782825Z * [new branch] gh/xush6528/18/base -> origin/gh/xush6528/18/base 2022-03-17T19:07:03.8784106Z * [new branch] gh/xush6528/18/head -> origin/gh/xush6528/18/head 2022-03-17T19:07:03.8786321Z * [new branch] gh/xush6528/54/base -> origin/gh/xush6528/54/base 2022-03-17T19:07:03.8787511Z * [new branch] gh/xush6528/54/head -> origin/gh/xush6528/54/head 2022-03-17T19:07:03.8789092Z * [new branch] gh/xush6528/54/orig -> origin/gh/xush6528/54/orig 2022-03-17T19:07:03.8790871Z * [new branch] gh/xush6528/58/base -> origin/gh/xush6528/58/base 2022-03-17T19:07:03.8792230Z * [new branch] gh/xush6528/58/head -> origin/gh/xush6528/58/head 2022-03-17T19:07:03.8794660Z * [new branch] gh/xuzhao9/1/base -> origin/gh/xuzhao9/1/base 2022-03-17T19:07:03.8795911Z * [new branch] gh/xuzhao9/1/orig -> origin/gh/xuzhao9/1/orig 2022-03-17T19:07:03.8797829Z * [new branch] gh/xuzhao9/2/base -> origin/gh/xuzhao9/2/base 2022-03-17T19:07:03.8799201Z * [new branch] gh/xuzhao9/2/orig -> origin/gh/xuzhao9/2/orig 2022-03-17T19:07:03.8800961Z * [new branch] gh/xuzhao9/3/base -> origin/gh/xuzhao9/3/base 2022-03-17T19:07:03.8802331Z * [new branch] gh/xuzhao9/3/orig -> origin/gh/xuzhao9/3/orig 2022-03-17T19:07:03.8804588Z * [new branch] gh/yf225/107/base -> origin/gh/yf225/107/base 2022-03-17T19:07:03.8806227Z * [new branch] gh/yf225/107/head -> origin/gh/yf225/107/head 2022-03-17T19:07:03.8806955Z * [new branch] gh/yf225/107/orig -> origin/gh/yf225/107/orig 2022-03-17T19:07:03.8808921Z * [new branch] gh/yf225/14/head -> origin/gh/yf225/14/head 2022-03-17T19:07:03.8810885Z * [new branch] gh/yf225/15/base -> origin/gh/yf225/15/base 2022-03-17T19:07:03.8811940Z * [new branch] gh/yf225/15/head -> origin/gh/yf225/15/head 2022-03-17T19:07:03.8813914Z * [new branch] gh/yf225/73/base -> origin/gh/yf225/73/base 2022-03-17T19:07:03.8815279Z * [new branch] gh/yf225/73/head -> origin/gh/yf225/73/head 2022-03-17T19:07:03.8816628Z * [new branch] gh/yf225/73/orig -> origin/gh/yf225/73/orig 2022-03-17T19:07:03.8818376Z * [new branch] gh/yf225/82/base -> origin/gh/yf225/82/base 2022-03-17T19:07:03.8819726Z * [new branch] gh/yf225/82/head -> origin/gh/yf225/82/head 2022-03-17T19:07:03.8821061Z * [new branch] gh/yf225/82/orig -> origin/gh/yf225/82/orig 2022-03-17T19:07:03.8823447Z * [new branch] gh/yf225/84/base -> origin/gh/yf225/84/base 2022-03-17T19:07:03.8825037Z * [new branch] gh/yf225/84/head -> origin/gh/yf225/84/head 2022-03-17T19:07:03.8826089Z * [new branch] gh/yf225/84/orig -> origin/gh/yf225/84/orig 2022-03-17T19:07:03.8827917Z * [new branch] gh/yf225/90/base -> origin/gh/yf225/90/base 2022-03-17T19:07:03.8829247Z * [new branch] gh/yf225/90/head -> origin/gh/yf225/90/head 2022-03-17T19:07:03.8830580Z * [new branch] gh/yf225/90/orig -> origin/gh/yf225/90/orig 2022-03-17T19:07:03.8832254Z * [new branch] gh/yf225/91/base -> origin/gh/yf225/91/base 2022-03-17T19:07:03.8833425Z * [new branch] gh/yf225/91/head -> origin/gh/yf225/91/head 2022-03-17T19:07:03.8834583Z * [new branch] gh/yf225/91/orig -> origin/gh/yf225/91/orig 2022-03-17T19:07:03.8836986Z * [new branch] gh/ysiraichi/31/base -> origin/gh/ysiraichi/31/base 2022-03-17T19:07:03.8838058Z * [new branch] gh/ysiraichi/31/head -> origin/gh/ysiraichi/31/head 2022-03-17T19:07:03.8839800Z * [new branch] gh/ysiraichi/31/orig -> origin/gh/ysiraichi/31/orig 2022-03-17T19:07:03.8841644Z * [new branch] gh/ysiraichi/35/base -> origin/gh/ysiraichi/35/base 2022-03-17T19:07:03.8842900Z * [new branch] gh/ysiraichi/35/head -> origin/gh/ysiraichi/35/head 2022-03-17T19:07:03.8844618Z * [new branch] gh/ysiraichi/35/orig -> origin/gh/ysiraichi/35/orig 2022-03-17T19:07:03.8846403Z * [new branch] gh/ysiraichi/36/base -> origin/gh/ysiraichi/36/base 2022-03-17T19:07:03.8848056Z * [new branch] gh/ysiraichi/36/orig -> origin/gh/ysiraichi/36/orig 2022-03-17T19:07:03.8850366Z * [new branch] gh/ysiraichi/37/base -> origin/gh/ysiraichi/37/base 2022-03-17T19:07:03.8851737Z * [new branch] gh/ysiraichi/37/head -> origin/gh/ysiraichi/37/head 2022-03-17T19:07:03.8853353Z * [new branch] gh/ysiraichi/37/orig -> origin/gh/ysiraichi/37/orig 2022-03-17T19:07:03.8855537Z * [new branch] gh/ysiraichi/38/base -> origin/gh/ysiraichi/38/base 2022-03-17T19:07:03.8856729Z * [new branch] gh/ysiraichi/38/head -> origin/gh/ysiraichi/38/head 2022-03-17T19:07:03.8858490Z * [new branch] gh/ysiraichi/38/orig -> origin/gh/ysiraichi/38/orig 2022-03-17T19:07:03.8859876Z * [new branch] gh/ysiraichi/39/base -> origin/gh/ysiraichi/39/base 2022-03-17T19:07:03.8861158Z * [new branch] gh/ysiraichi/39/head -> origin/gh/ysiraichi/39/head 2022-03-17T19:07:03.8863353Z * [new branch] gh/ysiraichi/39/orig -> origin/gh/ysiraichi/39/orig 2022-03-17T19:07:03.8865194Z * [new branch] gh/ysiraichi/40/base -> origin/gh/ysiraichi/40/base 2022-03-17T19:07:03.8866564Z * [new branch] gh/ysiraichi/40/head -> origin/gh/ysiraichi/40/head 2022-03-17T19:07:03.8868224Z * [new branch] gh/ysiraichi/40/orig -> origin/gh/ysiraichi/40/orig 2022-03-17T19:07:03.8870017Z * [new branch] gh/ysiraichi/41/base -> origin/gh/ysiraichi/41/base 2022-03-17T19:07:03.8871015Z * [new branch] gh/ysiraichi/41/head -> origin/gh/ysiraichi/41/head 2022-03-17T19:07:03.8872809Z * [new branch] gh/ysiraichi/41/orig -> origin/gh/ysiraichi/41/orig 2022-03-17T19:07:03.8875141Z * [new branch] gh/ysiraichi/42/base -> origin/gh/ysiraichi/42/base 2022-03-17T19:07:03.8876156Z * [new branch] gh/ysiraichi/42/head -> origin/gh/ysiraichi/42/head 2022-03-17T19:07:03.8877887Z * [new branch] gh/ysiraichi/42/orig -> origin/gh/ysiraichi/42/orig 2022-03-17T19:07:03.8879706Z * [new branch] gh/ysiraichi/43/base -> origin/gh/ysiraichi/43/base 2022-03-17T19:07:03.8880966Z * [new branch] gh/ysiraichi/43/head -> origin/gh/ysiraichi/43/head 2022-03-17T19:07:03.8882728Z * [new branch] gh/ysiraichi/43/orig -> origin/gh/ysiraichi/43/orig 2022-03-17T19:07:03.8884605Z * [new branch] gh/ysiraichi/44/base -> origin/gh/ysiraichi/44/base 2022-03-17T19:07:03.8886320Z * [new branch] gh/ysiraichi/44/head -> origin/gh/ysiraichi/44/head 2022-03-17T19:07:03.8887418Z * [new branch] gh/ysiraichi/44/orig -> origin/gh/ysiraichi/44/orig 2022-03-17T19:07:03.8890003Z * [new branch] gh/z-a-f/100/base -> origin/gh/z-a-f/100/base 2022-03-17T19:07:03.8891275Z * [new branch] gh/z-a-f/100/head -> origin/gh/z-a-f/100/head 2022-03-17T19:07:03.8893050Z * [new branch] gh/z-a-f/100/orig -> origin/gh/z-a-f/100/orig 2022-03-17T19:07:03.8894405Z * [new branch] gh/z-a-f/101/base -> origin/gh/z-a-f/101/base 2022-03-17T19:07:03.8895763Z * [new branch] gh/z-a-f/101/head -> origin/gh/z-a-f/101/head 2022-03-17T19:07:03.8897112Z * [new branch] gh/z-a-f/101/orig -> origin/gh/z-a-f/101/orig 2022-03-17T19:07:03.8898987Z * [new branch] gh/z-a-f/102/base -> origin/gh/z-a-f/102/base 2022-03-17T19:07:03.8900150Z * [new branch] gh/z-a-f/102/head -> origin/gh/z-a-f/102/head 2022-03-17T19:07:03.8903071Z * [new branch] gh/z-a-f/102/orig -> origin/gh/z-a-f/102/orig 2022-03-17T19:07:03.8904739Z * [new branch] gh/z-a-f/106/base -> origin/gh/z-a-f/106/base 2022-03-17T19:07:03.8906760Z * [new branch] gh/z-a-f/106/head -> origin/gh/z-a-f/106/head 2022-03-17T19:07:03.8908001Z * [new branch] gh/z-a-f/106/orig -> origin/gh/z-a-f/106/orig 2022-03-17T19:07:03.8910117Z * [new branch] gh/z-a-f/109/base -> origin/gh/z-a-f/109/base 2022-03-17T19:07:03.8912156Z * [new branch] gh/z-a-f/109/head -> origin/gh/z-a-f/109/head 2022-03-17T19:07:03.8913193Z * [new branch] gh/z-a-f/109/orig -> origin/gh/z-a-f/109/orig 2022-03-17T19:07:03.8915428Z * [new branch] gh/z-a-f/112/base -> origin/gh/z-a-f/112/base 2022-03-17T19:07:03.8916797Z * [new branch] gh/z-a-f/112/head -> origin/gh/z-a-f/112/head 2022-03-17T19:07:03.8918139Z * [new branch] gh/z-a-f/112/orig -> origin/gh/z-a-f/112/orig 2022-03-17T19:07:03.8920123Z * [new branch] gh/z-a-f/113/base -> origin/gh/z-a-f/113/base 2022-03-17T19:07:03.8921884Z * [new branch] gh/z-a-f/113/head -> origin/gh/z-a-f/113/head 2022-03-17T19:07:03.8923676Z * [new branch] gh/z-a-f/121/base -> origin/gh/z-a-f/121/base 2022-03-17T19:07:03.8924865Z * [new branch] gh/z-a-f/121/head -> origin/gh/z-a-f/121/head 2022-03-17T19:07:03.8926253Z * [new branch] gh/z-a-f/121/orig -> origin/gh/z-a-f/121/orig 2022-03-17T19:07:03.8928380Z * [new branch] gh/z-a-f/122/base -> origin/gh/z-a-f/122/base 2022-03-17T19:07:03.8929652Z * [new branch] gh/z-a-f/122/head -> origin/gh/z-a-f/122/head 2022-03-17T19:07:03.8931587Z * [new branch] gh/z-a-f/122/orig -> origin/gh/z-a-f/122/orig 2022-03-17T19:07:03.8933427Z * [new branch] gh/z-a-f/123/base -> origin/gh/z-a-f/123/base 2022-03-17T19:07:03.8934585Z * [new branch] gh/z-a-f/123/head -> origin/gh/z-a-f/123/head 2022-03-17T19:07:03.8935949Z * [new branch] gh/z-a-f/123/orig -> origin/gh/z-a-f/123/orig 2022-03-17T19:07:03.8938107Z * [new branch] gh/z-a-f/124/base -> origin/gh/z-a-f/124/base 2022-03-17T19:07:03.8939274Z * [new branch] gh/z-a-f/124/head -> origin/gh/z-a-f/124/head 2022-03-17T19:07:03.8941175Z * [new branch] gh/z-a-f/124/orig -> origin/gh/z-a-f/124/orig 2022-03-17T19:07:03.8943416Z * [new branch] gh/z-a-f/127/base -> origin/gh/z-a-f/127/base 2022-03-17T19:07:03.8944669Z * [new branch] gh/z-a-f/127/head -> origin/gh/z-a-f/127/head 2022-03-17T19:07:03.8946179Z * [new branch] gh/z-a-f/127/orig -> origin/gh/z-a-f/127/orig 2022-03-17T19:07:03.8948208Z * [new branch] gh/z-a-f/129/base -> origin/gh/z-a-f/129/base 2022-03-17T19:07:03.8949406Z * [new branch] gh/z-a-f/129/head -> origin/gh/z-a-f/129/head 2022-03-17T19:07:03.8950946Z * [new branch] gh/z-a-f/129/orig -> origin/gh/z-a-f/129/orig 2022-03-17T19:07:03.8952907Z * [new branch] gh/z-a-f/130/base -> origin/gh/z-a-f/130/base 2022-03-17T19:07:03.8954865Z * [new branch] gh/z-a-f/130/head -> origin/gh/z-a-f/130/head 2022-03-17T19:07:03.8956222Z * [new branch] gh/z-a-f/130/orig -> origin/gh/z-a-f/130/orig 2022-03-17T19:07:03.8958184Z * [new branch] gh/z-a-f/131/base -> origin/gh/z-a-f/131/base 2022-03-17T19:07:03.8959366Z * [new branch] gh/z-a-f/131/head -> origin/gh/z-a-f/131/head 2022-03-17T19:07:03.8961112Z * [new branch] gh/z-a-f/131/orig -> origin/gh/z-a-f/131/orig 2022-03-17T19:07:03.8962976Z * [new branch] gh/z-a-f/30/base -> origin/gh/z-a-f/30/base 2022-03-17T19:07:03.8964131Z * [new branch] gh/z-a-f/30/head -> origin/gh/z-a-f/30/head 2022-03-17T19:07:03.8965730Z * [new branch] gh/z-a-f/30/orig -> origin/gh/z-a-f/30/orig 2022-03-17T19:07:03.8967656Z * [new branch] gh/z-a-f/44/base -> origin/gh/z-a-f/44/base 2022-03-17T19:07:03.8969024Z * [new branch] gh/z-a-f/44/head -> origin/gh/z-a-f/44/head 2022-03-17T19:07:03.8970644Z * [new branch] gh/z-a-f/44/orig -> origin/gh/z-a-f/44/orig 2022-03-17T19:07:03.8972656Z * [new branch] gh/z-a-f/45/base -> origin/gh/z-a-f/45/base 2022-03-17T19:07:03.8974066Z * [new branch] gh/z-a-f/45/head -> origin/gh/z-a-f/45/head 2022-03-17T19:07:03.8975428Z * [new branch] gh/z-a-f/45/orig -> origin/gh/z-a-f/45/orig 2022-03-17T19:07:03.8977168Z * [new branch] gh/z-a-f/46/base -> origin/gh/z-a-f/46/base 2022-03-17T19:07:03.8978414Z * [new branch] gh/z-a-f/46/head -> origin/gh/z-a-f/46/head 2022-03-17T19:07:03.8980096Z * [new branch] gh/z-a-f/46/orig -> origin/gh/z-a-f/46/orig 2022-03-17T19:07:03.8981999Z * [new branch] gh/z-a-f/47/base -> origin/gh/z-a-f/47/base 2022-03-17T19:07:03.8983684Z * [new branch] gh/z-a-f/47/head -> origin/gh/z-a-f/47/head 2022-03-17T19:07:03.8984823Z * [new branch] gh/z-a-f/47/orig -> origin/gh/z-a-f/47/orig 2022-03-17T19:07:03.8986843Z * [new branch] gh/z-a-f/52/base -> origin/gh/z-a-f/52/base 2022-03-17T19:07:03.8988102Z * [new branch] gh/z-a-f/52/head -> origin/gh/z-a-f/52/head 2022-03-17T19:07:03.8989687Z * [new branch] gh/z-a-f/52/orig -> origin/gh/z-a-f/52/orig 2022-03-17T19:07:03.8991452Z * [new branch] gh/z-a-f/56/base -> origin/gh/z-a-f/56/base 2022-03-17T19:07:03.8993160Z * [new branch] gh/z-a-f/56/head -> origin/gh/z-a-f/56/head 2022-03-17T19:07:03.8994379Z * [new branch] gh/z-a-f/56/orig -> origin/gh/z-a-f/56/orig 2022-03-17T19:07:03.8996203Z * [new branch] gh/z-a-f/57/base -> origin/gh/z-a-f/57/base 2022-03-17T19:07:03.8997378Z * [new branch] gh/z-a-f/57/head -> origin/gh/z-a-f/57/head 2022-03-17T19:07:03.8999128Z * [new branch] gh/z-a-f/57/orig -> origin/gh/z-a-f/57/orig 2022-03-17T19:07:03.9000747Z * [new branch] gh/z-a-f/58/base -> origin/gh/z-a-f/58/base 2022-03-17T19:07:03.9002104Z * [new branch] gh/z-a-f/58/head -> origin/gh/z-a-f/58/head 2022-03-17T19:07:03.9003660Z * [new branch] gh/z-a-f/58/orig -> origin/gh/z-a-f/58/orig 2022-03-17T19:07:03.9005484Z * [new branch] gh/z-a-f/70/base -> origin/gh/z-a-f/70/base 2022-03-17T19:07:03.9006641Z * [new branch] gh/z-a-f/70/head -> origin/gh/z-a-f/70/head 2022-03-17T19:07:03.9008277Z * [new branch] gh/z-a-f/70/orig -> origin/gh/z-a-f/70/orig 2022-03-17T19:07:03.9010102Z * [new branch] gh/z-a-f/72/base -> origin/gh/z-a-f/72/base 2022-03-17T19:07:03.9011301Z * [new branch] gh/z-a-f/72/head -> origin/gh/z-a-f/72/head 2022-03-17T19:07:03.9013262Z * [new branch] gh/z-a-f/72/orig -> origin/gh/z-a-f/72/orig 2022-03-17T19:07:03.9014833Z * [new branch] gh/z-a-f/73/base -> origin/gh/z-a-f/73/base 2022-03-17T19:07:03.9016082Z * [new branch] gh/z-a-f/73/head -> origin/gh/z-a-f/73/head 2022-03-17T19:07:03.9017652Z * [new branch] gh/z-a-f/73/orig -> origin/gh/z-a-f/73/orig 2022-03-17T19:07:03.9019403Z * [new branch] gh/z-a-f/74/base -> origin/gh/z-a-f/74/base 2022-03-17T19:07:03.9020669Z * [new branch] gh/z-a-f/74/head -> origin/gh/z-a-f/74/head 2022-03-17T19:07:03.9022974Z * [new branch] gh/z-a-f/74/orig -> origin/gh/z-a-f/74/orig 2022-03-17T19:07:03.9024290Z * [new branch] gh/z-a-f/76/base -> origin/gh/z-a-f/76/base 2022-03-17T19:07:03.9025596Z * [new branch] gh/z-a-f/76/head -> origin/gh/z-a-f/76/head 2022-03-17T19:07:03.9026988Z * [new branch] gh/z-a-f/76/orig -> origin/gh/z-a-f/76/orig 2022-03-17T19:07:03.9029219Z * [new branch] gh/z-a-f/77/base -> origin/gh/z-a-f/77/base 2022-03-17T19:07:03.9030333Z * [new branch] gh/z-a-f/77/head -> origin/gh/z-a-f/77/head 2022-03-17T19:07:03.9031722Z * [new branch] gh/z-a-f/77/orig -> origin/gh/z-a-f/77/orig 2022-03-17T19:07:03.9033749Z * [new branch] gh/z-a-f/78/base -> origin/gh/z-a-f/78/base 2022-03-17T19:07:03.9034844Z * [new branch] gh/z-a-f/78/head -> origin/gh/z-a-f/78/head 2022-03-17T19:07:03.9036188Z * [new branch] gh/z-a-f/78/orig -> origin/gh/z-a-f/78/orig 2022-03-17T19:07:03.9038891Z * [new branch] gh/z-a-f/80/base -> origin/gh/z-a-f/80/base 2022-03-17T19:07:03.9040017Z * [new branch] gh/z-a-f/80/head -> origin/gh/z-a-f/80/head 2022-03-17T19:07:03.9041352Z * [new branch] gh/z-a-f/80/orig -> origin/gh/z-a-f/80/orig 2022-03-17T19:07:03.9043664Z * [new branch] gh/z-a-f/81/base -> origin/gh/z-a-f/81/base 2022-03-17T19:07:03.9044881Z * [new branch] gh/z-a-f/81/head -> origin/gh/z-a-f/81/head 2022-03-17T19:07:03.9046236Z * [new branch] gh/z-a-f/81/orig -> origin/gh/z-a-f/81/orig 2022-03-17T19:07:03.9048379Z * [new branch] gh/z-a-f/82/base -> origin/gh/z-a-f/82/base 2022-03-17T19:07:03.9049523Z * [new branch] gh/z-a-f/82/head -> origin/gh/z-a-f/82/head 2022-03-17T19:07:03.9051269Z * [new branch] gh/z-a-f/82/orig -> origin/gh/z-a-f/82/orig 2022-03-17T19:07:03.9052891Z * [new branch] gh/z-a-f/83/base -> origin/gh/z-a-f/83/base 2022-03-17T19:07:03.9054133Z * [new branch] gh/z-a-f/83/head -> origin/gh/z-a-f/83/head 2022-03-17T19:07:03.9055682Z * [new branch] gh/z-a-f/83/orig -> origin/gh/z-a-f/83/orig 2022-03-17T19:07:03.9057637Z * [new branch] gh/z-a-f/84/base -> origin/gh/z-a-f/84/base 2022-03-17T19:07:03.9058808Z * [new branch] gh/z-a-f/84/head -> origin/gh/z-a-f/84/head 2022-03-17T19:07:03.9060440Z * [new branch] gh/z-a-f/84/orig -> origin/gh/z-a-f/84/orig 2022-03-17T19:07:03.9063506Z * [new branch] gh/z-a-f/87/base -> origin/gh/z-a-f/87/base 2022-03-17T19:07:03.9064965Z * [new branch] gh/z-a-f/87/head -> origin/gh/z-a-f/87/head 2022-03-17T19:07:03.9066203Z * [new branch] gh/z-a-f/87/orig -> origin/gh/z-a-f/87/orig 2022-03-17T19:07:03.9067940Z * [new branch] gh/z-a-f/88/base -> origin/gh/z-a-f/88/base 2022-03-17T19:07:03.9068871Z * [new branch] gh/z-a-f/88/head -> origin/gh/z-a-f/88/head 2022-03-17T19:07:03.9070613Z * [new branch] gh/z-a-f/88/orig -> origin/gh/z-a-f/88/orig 2022-03-17T19:07:03.9071972Z * [new branch] gh/z-a-f/89/base -> origin/gh/z-a-f/89/base 2022-03-17T19:07:03.9073675Z * [new branch] gh/z-a-f/89/head -> origin/gh/z-a-f/89/head 2022-03-17T19:07:03.9074853Z * [new branch] gh/z-a-f/89/orig -> origin/gh/z-a-f/89/orig 2022-03-17T19:07:03.9077172Z * [new branch] gh/zafartahirov/1/base -> origin/gh/zafartahirov/1/base 2022-03-17T19:07:03.9079147Z * [new branch] gh/zafartahirov/101/base -> origin/gh/zafartahirov/101/base 2022-03-17T19:07:03.9080669Z * [new branch] gh/zafartahirov/101/head -> origin/gh/zafartahirov/101/head 2022-03-17T19:07:03.9082389Z * [new branch] gh/zafartahirov/101/orig -> origin/gh/zafartahirov/101/orig 2022-03-17T19:07:03.9084062Z * [new branch] gh/zafartahirov/104/base -> origin/gh/zafartahirov/104/base 2022-03-17T19:07:03.9085440Z * [new branch] gh/zafartahirov/104/orig -> origin/gh/zafartahirov/104/orig 2022-03-17T19:07:03.9087243Z * [new branch] gh/zafartahirov/106/base -> origin/gh/zafartahirov/106/base 2022-03-17T19:07:03.9088442Z * [new branch] gh/zafartahirov/106/head -> origin/gh/zafartahirov/106/head 2022-03-17T19:07:03.9090139Z * [new branch] gh/zafartahirov/106/orig -> origin/gh/zafartahirov/106/orig 2022-03-17T19:07:03.9091932Z * [new branch] gh/zafartahirov/114/base -> origin/gh/zafartahirov/114/base 2022-03-17T19:07:03.9093562Z * [new branch] gh/zafartahirov/114/head -> origin/gh/zafartahirov/114/head 2022-03-17T19:07:03.9094706Z * [new branch] gh/zafartahirov/115/base -> origin/gh/zafartahirov/115/base 2022-03-17T19:07:03.9096562Z * [new branch] gh/zafartahirov/115/head -> origin/gh/zafartahirov/115/head 2022-03-17T19:07:03.9097784Z * [new branch] gh/zafartahirov/116/base -> origin/gh/zafartahirov/116/base 2022-03-17T19:07:03.9099172Z * [new branch] gh/zafartahirov/116/head -> origin/gh/zafartahirov/116/head 2022-03-17T19:07:03.9101230Z * [new branch] gh/zafartahirov/118/base -> origin/gh/zafartahirov/118/base 2022-03-17T19:07:03.9102922Z * [new branch] gh/zafartahirov/118/head -> origin/gh/zafartahirov/118/head 2022-03-17T19:07:03.9104541Z * [new branch] gh/zafartahirov/118/orig -> origin/gh/zafartahirov/118/orig 2022-03-17T19:07:03.9106253Z * [new branch] gh/zafartahirov/119/base -> origin/gh/zafartahirov/119/base 2022-03-17T19:07:03.9108202Z * [new branch] gh/zafartahirov/119/head -> origin/gh/zafartahirov/119/head 2022-03-17T19:07:03.9109386Z * [new branch] gh/zafartahirov/119/orig -> origin/gh/zafartahirov/119/orig 2022-03-17T19:07:03.9111344Z * [new branch] gh/zafartahirov/120/base -> origin/gh/zafartahirov/120/base 2022-03-17T19:07:03.9112537Z * [new branch] gh/zafartahirov/120/head -> origin/gh/zafartahirov/120/head 2022-03-17T19:07:03.9114123Z * [new branch] gh/zafartahirov/120/orig -> origin/gh/zafartahirov/120/orig 2022-03-17T19:07:03.9115787Z * [new branch] gh/zafartahirov/121/base -> origin/gh/zafartahirov/121/base 2022-03-17T19:07:03.9116779Z * [new branch] gh/zafartahirov/121/head -> origin/gh/zafartahirov/121/head 2022-03-17T19:07:03.9118503Z * [new branch] gh/zafartahirov/121/orig -> origin/gh/zafartahirov/121/orig 2022-03-17T19:07:03.9120011Z * [new branch] gh/zafartahirov/122/base -> origin/gh/zafartahirov/122/base 2022-03-17T19:07:03.9121159Z * [new branch] gh/zafartahirov/122/head -> origin/gh/zafartahirov/122/head 2022-03-17T19:07:03.9122831Z * [new branch] gh/zafartahirov/122/orig -> origin/gh/zafartahirov/122/orig 2022-03-17T19:07:03.9124632Z * [new branch] gh/zafartahirov/123/base -> origin/gh/zafartahirov/123/base 2022-03-17T19:07:03.9125780Z * [new branch] gh/zafartahirov/123/head -> origin/gh/zafartahirov/123/head 2022-03-17T19:07:03.9127934Z * [new branch] gh/zafartahirov/124/base -> origin/gh/zafartahirov/124/base 2022-03-17T19:07:03.9129092Z * [new branch] gh/zafartahirov/124/head -> origin/gh/zafartahirov/124/head 2022-03-17T19:07:03.9131065Z * [new branch] gh/zafartahirov/124/orig -> origin/gh/zafartahirov/124/orig 2022-03-17T19:07:03.9132885Z * [new branch] gh/zafartahirov/125/base -> origin/gh/zafartahirov/125/base 2022-03-17T19:07:03.9134058Z * [new branch] gh/zafartahirov/125/head -> origin/gh/zafartahirov/125/head 2022-03-17T19:07:03.9135737Z * [new branch] gh/zafartahirov/125/orig -> origin/gh/zafartahirov/125/orig 2022-03-17T19:07:03.9137715Z * [new branch] gh/zafartahirov/2/base -> origin/gh/zafartahirov/2/base 2022-03-17T19:07:03.9139341Z * [new branch] gh/zafartahirov/61/base -> origin/gh/zafartahirov/61/base 2022-03-17T19:07:03.9140717Z * [new branch] gh/zafartahirov/61/head -> origin/gh/zafartahirov/61/head 2022-03-17T19:07:03.9142769Z * [new branch] gh/zafartahirov/61/orig -> origin/gh/zafartahirov/61/orig 2022-03-17T19:07:03.9145100Z * [new branch] gh/zasdfgbnm/10/base -> origin/gh/zasdfgbnm/10/base 2022-03-17T19:07:03.9147259Z * [new branch] gh/zasdfgbnm/10/head -> origin/gh/zasdfgbnm/10/head 2022-03-17T19:07:03.9148320Z * [new branch] gh/zasdfgbnm/10/orig -> origin/gh/zasdfgbnm/10/orig 2022-03-17T19:07:03.9150909Z * [new branch] gh/zasdfgbnm/14/base -> origin/gh/zasdfgbnm/14/base 2022-03-17T19:07:03.9152099Z * [new branch] gh/zasdfgbnm/14/orig -> origin/gh/zasdfgbnm/14/orig 2022-03-17T19:07:03.9154612Z * [new branch] gh/zasdfgbnm/15/base -> origin/gh/zasdfgbnm/15/base 2022-03-17T19:07:03.9155778Z * [new branch] gh/zasdfgbnm/15/orig -> origin/gh/zasdfgbnm/15/orig 2022-03-17T19:07:03.9157724Z * [new branch] gh/zasdfgbnm/16/base -> origin/gh/zasdfgbnm/16/base 2022-03-17T19:07:03.9158888Z * [new branch] gh/zasdfgbnm/16/orig -> origin/gh/zasdfgbnm/16/orig 2022-03-17T19:07:03.9160805Z * [new branch] gh/zasdfgbnm/17/orig -> origin/gh/zasdfgbnm/17/orig 2022-03-17T19:07:03.9162493Z * [new branch] gh/zasdfgbnm/18/base -> origin/gh/zasdfgbnm/18/base 2022-03-17T19:07:03.9163660Z * [new branch] gh/zasdfgbnm/18/orig -> origin/gh/zasdfgbnm/18/orig 2022-03-17T19:07:03.9165572Z * [new branch] gh/zasdfgbnm/19/orig -> origin/gh/zasdfgbnm/19/orig 2022-03-17T19:07:03.9167338Z * [new branch] gh/zasdfgbnm/20/base -> origin/gh/zasdfgbnm/20/base 2022-03-17T19:07:03.9168953Z * [new branch] gh/zasdfgbnm/20/orig -> origin/gh/zasdfgbnm/20/orig 2022-03-17T19:07:03.9171291Z * [new branch] gh/zasdfgbnm/21/base -> origin/gh/zasdfgbnm/21/base 2022-03-17T19:07:03.9172580Z * [new branch] gh/zasdfgbnm/21/head -> origin/gh/zasdfgbnm/21/head 2022-03-17T19:07:03.9174290Z * [new branch] gh/zasdfgbnm/21/orig -> origin/gh/zasdfgbnm/21/orig 2022-03-17T19:07:03.9175541Z * [new branch] gh/zasdfgbnm/22/base -> origin/gh/zasdfgbnm/22/base 2022-03-17T19:07:03.9177342Z * [new branch] gh/zasdfgbnm/22/head -> origin/gh/zasdfgbnm/22/head 2022-03-17T19:07:03.9178614Z * [new branch] gh/zasdfgbnm/22/orig -> origin/gh/zasdfgbnm/22/orig 2022-03-17T19:07:03.9180033Z * [new branch] gh/zasdfgbnm/23/base -> origin/gh/zasdfgbnm/23/base 2022-03-17T19:07:03.9181365Z * [new branch] gh/zasdfgbnm/23/head -> origin/gh/zasdfgbnm/23/head 2022-03-17T19:07:03.9183392Z * [new branch] gh/zasdfgbnm/23/orig -> origin/gh/zasdfgbnm/23/orig 2022-03-17T19:07:03.9185328Z * [new branch] gh/zasdfgbnm/24/base -> origin/gh/zasdfgbnm/24/base 2022-03-17T19:07:03.9186377Z * [new branch] gh/zasdfgbnm/24/orig -> origin/gh/zasdfgbnm/24/orig 2022-03-17T19:07:03.9189301Z * [new branch] gh/zasdfgbnm/25/base -> origin/gh/zasdfgbnm/25/base 2022-03-17T19:07:03.9190364Z * [new branch] gh/zasdfgbnm/25/orig -> origin/gh/zasdfgbnm/25/orig 2022-03-17T19:07:03.9192398Z * [new branch] gh/zasdfgbnm/26/orig -> origin/gh/zasdfgbnm/26/orig 2022-03-17T19:07:03.9194077Z * [new branch] gh/zasdfgbnm/27/orig -> origin/gh/zasdfgbnm/27/orig 2022-03-17T19:07:03.9195780Z * [new branch] gh/zasdfgbnm/28/base -> origin/gh/zasdfgbnm/28/base 2022-03-17T19:07:03.9196921Z * [new branch] gh/zasdfgbnm/28/orig -> origin/gh/zasdfgbnm/28/orig 2022-03-17T19:07:03.9199042Z * [new branch] gh/zasdfgbnm/29/base -> origin/gh/zasdfgbnm/29/base 2022-03-17T19:07:03.9200074Z * [new branch] gh/zasdfgbnm/29/orig -> origin/gh/zasdfgbnm/29/orig 2022-03-17T19:07:03.9202062Z * [new branch] gh/zasdfgbnm/30/base -> origin/gh/zasdfgbnm/30/base 2022-03-17T19:07:03.9203230Z * [new branch] gh/zasdfgbnm/30/orig -> origin/gh/zasdfgbnm/30/orig 2022-03-17T19:07:03.9205752Z * [new branch] gh/zasdfgbnm/31/orig -> origin/gh/zasdfgbnm/31/orig 2022-03-17T19:07:03.9207744Z * [new branch] gh/zasdfgbnm/32/base -> origin/gh/zasdfgbnm/32/base 2022-03-17T19:07:03.9209341Z * [new branch] gh/zasdfgbnm/32/orig -> origin/gh/zasdfgbnm/32/orig 2022-03-17T19:07:03.9211139Z * [new branch] gh/zasdfgbnm/33/base -> origin/gh/zasdfgbnm/33/base 2022-03-17T19:07:03.9212396Z * [new branch] gh/zasdfgbnm/33/orig -> origin/gh/zasdfgbnm/33/orig 2022-03-17T19:07:03.9214383Z * [new branch] gh/zasdfgbnm/34/orig -> origin/gh/zasdfgbnm/34/orig 2022-03-17T19:07:03.9215757Z * [new branch] gh/zasdfgbnm/35/base -> origin/gh/zasdfgbnm/35/base 2022-03-17T19:07:03.9217365Z * [new branch] gh/zasdfgbnm/35/head -> origin/gh/zasdfgbnm/35/head 2022-03-17T19:07:03.9218613Z * [new branch] gh/zasdfgbnm/35/orig -> origin/gh/zasdfgbnm/35/orig 2022-03-17T19:07:03.9220786Z * [new branch] gh/zasdfgbnm/36/base -> origin/gh/zasdfgbnm/36/base 2022-03-17T19:07:03.9222563Z * [new branch] gh/zasdfgbnm/36/orig -> origin/gh/zasdfgbnm/36/orig 2022-03-17T19:07:03.9224736Z * [new branch] gh/zasdfgbnm/37/base -> origin/gh/zasdfgbnm/37/base 2022-03-17T19:07:03.9226285Z * [new branch] gh/zasdfgbnm/37/orig -> origin/gh/zasdfgbnm/37/orig 2022-03-17T19:07:03.9228028Z * [new branch] gh/zasdfgbnm/38/base -> origin/gh/zasdfgbnm/38/base 2022-03-17T19:07:03.9229209Z * [new branch] gh/zasdfgbnm/38/orig -> origin/gh/zasdfgbnm/38/orig 2022-03-17T19:07:03.9231345Z * [new branch] gh/zasdfgbnm/39/base -> origin/gh/zasdfgbnm/39/base 2022-03-17T19:07:03.9233258Z * [new branch] gh/zasdfgbnm/39/orig -> origin/gh/zasdfgbnm/39/orig 2022-03-17T19:07:03.9235017Z * [new branch] gh/zasdfgbnm/40/base -> origin/gh/zasdfgbnm/40/base 2022-03-17T19:07:03.9236195Z * [new branch] gh/zasdfgbnm/40/orig -> origin/gh/zasdfgbnm/40/orig 2022-03-17T19:07:03.9238281Z * [new branch] gh/zasdfgbnm/41/base -> origin/gh/zasdfgbnm/41/base 2022-03-17T19:07:03.9239328Z * [new branch] gh/zasdfgbnm/41/orig -> origin/gh/zasdfgbnm/41/orig 2022-03-17T19:07:03.9241374Z * [new branch] gh/zasdfgbnm/42/base -> origin/gh/zasdfgbnm/42/base 2022-03-17T19:07:03.9242749Z * [new branch] gh/zasdfgbnm/42/orig -> origin/gh/zasdfgbnm/42/orig 2022-03-17T19:07:03.9244872Z * [new branch] gh/zasdfgbnm/43/base -> origin/gh/zasdfgbnm/43/base 2022-03-17T19:07:03.9246558Z * [new branch] gh/zasdfgbnm/44/orig -> origin/gh/zasdfgbnm/44/orig 2022-03-17T19:07:03.9248288Z * [new branch] gh/zasdfgbnm/45/base -> origin/gh/zasdfgbnm/45/base 2022-03-17T19:07:03.9249713Z * [new branch] gh/zasdfgbnm/45/orig -> origin/gh/zasdfgbnm/45/orig 2022-03-17T19:07:03.9252012Z * [new branch] gh/zasdfgbnm/47/base -> origin/gh/zasdfgbnm/47/base 2022-03-17T19:07:03.9253174Z * [new branch] gh/zasdfgbnm/47/orig -> origin/gh/zasdfgbnm/47/orig 2022-03-17T19:07:03.9255154Z * [new branch] gh/zasdfgbnm/48/base -> origin/gh/zasdfgbnm/48/base 2022-03-17T19:07:03.9256278Z * [new branch] gh/zasdfgbnm/48/orig -> origin/gh/zasdfgbnm/48/orig 2022-03-17T19:07:03.9258651Z * [new branch] gh/zasdfgbnm/49/base -> origin/gh/zasdfgbnm/49/base 2022-03-17T19:07:03.9259822Z * [new branch] gh/zasdfgbnm/49/orig -> origin/gh/zasdfgbnm/49/orig 2022-03-17T19:07:03.9263748Z * [new branch] gh/zasdfgbnm/50/base -> origin/gh/zasdfgbnm/50/base 2022-03-17T19:07:03.9265409Z * [new branch] gh/zasdfgbnm/50/orig -> origin/gh/zasdfgbnm/50/orig 2022-03-17T19:07:03.9267219Z * [new branch] gh/zasdfgbnm/51/base -> origin/gh/zasdfgbnm/51/base 2022-03-17T19:07:03.9268385Z * [new branch] gh/zasdfgbnm/51/orig -> origin/gh/zasdfgbnm/51/orig 2022-03-17T19:07:03.9270388Z * [new branch] gh/zasdfgbnm/52/orig -> origin/gh/zasdfgbnm/52/orig 2022-03-17T19:07:03.9272093Z * [new branch] gh/zasdfgbnm/53/orig -> origin/gh/zasdfgbnm/53/orig 2022-03-17T19:07:03.9273803Z * [new branch] gh/zasdfgbnm/54/orig -> origin/gh/zasdfgbnm/54/orig 2022-03-17T19:07:03.9275597Z * [new branch] gh/zasdfgbnm/55/orig -> origin/gh/zasdfgbnm/55/orig 2022-03-17T19:07:03.9277196Z * [new branch] gh/zasdfgbnm/56/orig -> origin/gh/zasdfgbnm/56/orig 2022-03-17T19:07:03.9278905Z * [new branch] gh/zasdfgbnm/57/orig -> origin/gh/zasdfgbnm/57/orig 2022-03-17T19:07:03.9280797Z * [new branch] gh/zasdfgbnm/58/orig -> origin/gh/zasdfgbnm/58/orig 2022-03-17T19:07:03.9283032Z * [new branch] gh/zasdfgbnm/59/orig -> origin/gh/zasdfgbnm/59/orig 2022-03-17T19:07:03.9284769Z * [new branch] gh/zasdfgbnm/60/orig -> origin/gh/zasdfgbnm/60/orig 2022-03-17T19:07:03.9286986Z * [new branch] gh/zasdfgbnm/61/orig -> origin/gh/zasdfgbnm/61/orig 2022-03-17T19:07:03.9288850Z * [new branch] gh/zasdfgbnm/62/base -> origin/gh/zasdfgbnm/62/base 2022-03-17T19:07:03.9289980Z * [new branch] gh/zasdfgbnm/62/orig -> origin/gh/zasdfgbnm/62/orig 2022-03-17T19:07:03.9291946Z * [new branch] gh/zasdfgbnm/63/orig -> origin/gh/zasdfgbnm/63/orig 2022-03-17T19:07:03.9294158Z * [new branch] gh/zasdfgbnm/64/orig -> origin/gh/zasdfgbnm/64/orig 2022-03-17T19:07:03.9295921Z * [new branch] gh/zasdfgbnm/65/orig -> origin/gh/zasdfgbnm/65/orig 2022-03-17T19:07:03.9297841Z * [new branch] gh/zasdfgbnm/66/base -> origin/gh/zasdfgbnm/66/base 2022-03-17T19:07:03.9298889Z * [new branch] gh/zasdfgbnm/66/orig -> origin/gh/zasdfgbnm/66/orig 2022-03-17T19:07:03.9301074Z * [new branch] gh/zasdfgbnm/67/orig -> origin/gh/zasdfgbnm/67/orig 2022-03-17T19:07:03.9303655Z * [new branch] gh/zasdfgbnm/68/base -> origin/gh/zasdfgbnm/68/base 2022-03-17T19:07:03.9304734Z * [new branch] gh/zasdfgbnm/68/orig -> origin/gh/zasdfgbnm/68/orig 2022-03-17T19:07:03.9306612Z * [new branch] gh/zasdfgbnm/69/orig -> origin/gh/zasdfgbnm/69/orig 2022-03-17T19:07:03.9308327Z * [new branch] gh/zasdfgbnm/70/orig -> origin/gh/zasdfgbnm/70/orig 2022-03-17T19:07:03.9310053Z * [new branch] gh/zasdfgbnm/71/orig -> origin/gh/zasdfgbnm/71/orig 2022-03-17T19:07:03.9311822Z * [new branch] gh/zasdfgbnm/72/base -> origin/gh/zasdfgbnm/72/base 2022-03-17T19:07:03.9313021Z * [new branch] gh/zasdfgbnm/72/orig -> origin/gh/zasdfgbnm/72/orig 2022-03-17T19:07:03.9315254Z * [new branch] gh/zasdfgbnm/73/base -> origin/gh/zasdfgbnm/73/base 2022-03-17T19:07:03.9316456Z * [new branch] gh/zasdfgbnm/73/orig -> origin/gh/zasdfgbnm/73/orig 2022-03-17T19:07:03.9318522Z * [new branch] gh/zasdfgbnm/74/base -> origin/gh/zasdfgbnm/74/base 2022-03-17T19:07:03.9319712Z * [new branch] gh/zasdfgbnm/74/orig -> origin/gh/zasdfgbnm/74/orig 2022-03-17T19:07:03.9321742Z * [new branch] gh/zasdfgbnm/76/base -> origin/gh/zasdfgbnm/76/base 2022-03-17T19:07:03.9323164Z * [new branch] gh/zasdfgbnm/76/orig -> origin/gh/zasdfgbnm/76/orig 2022-03-17T19:07:03.9325067Z * [new branch] gh/zasdfgbnm/77/base -> origin/gh/zasdfgbnm/77/base 2022-03-17T19:07:03.9326270Z * [new branch] gh/zasdfgbnm/77/orig -> origin/gh/zasdfgbnm/77/orig 2022-03-17T19:07:03.9328819Z * [new branch] gh/zasdfgbnm/78/orig -> origin/gh/zasdfgbnm/78/orig 2022-03-17T19:07:03.9330629Z * [new branch] gh/zasdfgbnm/79/orig -> origin/gh/zasdfgbnm/79/orig 2022-03-17T19:07:03.9332643Z * [new branch] gh/zasdfgbnm/8/base -> origin/gh/zasdfgbnm/8/base 2022-03-17T19:07:03.9334272Z * [new branch] gh/zasdfgbnm/8/head -> origin/gh/zasdfgbnm/8/head 2022-03-17T19:07:03.9335542Z * [new branch] gh/zasdfgbnm/8/orig -> origin/gh/zasdfgbnm/8/orig 2022-03-17T19:07:03.9337492Z * [new branch] gh/zasdfgbnm/80/base -> origin/gh/zasdfgbnm/80/base 2022-03-17T19:07:03.9338904Z * [new branch] gh/zasdfgbnm/80/orig -> origin/gh/zasdfgbnm/80/orig 2022-03-17T19:07:03.9340792Z * [new branch] gh/zasdfgbnm/81/base -> origin/gh/zasdfgbnm/81/base 2022-03-17T19:07:03.9342431Z * [new branch] gh/zasdfgbnm/81/orig -> origin/gh/zasdfgbnm/81/orig 2022-03-17T19:07:03.9344264Z * [new branch] gh/zasdfgbnm/82/orig -> origin/gh/zasdfgbnm/82/orig 2022-03-17T19:07:03.9346108Z * [new branch] gh/zasdfgbnm/83/base -> origin/gh/zasdfgbnm/83/base 2022-03-17T19:07:03.9347514Z * [new branch] gh/zasdfgbnm/83/orig -> origin/gh/zasdfgbnm/83/orig 2022-03-17T19:07:03.9349378Z * [new branch] gh/zasdfgbnm/84/orig -> origin/gh/zasdfgbnm/84/orig 2022-03-17T19:07:03.9351171Z * [new branch] gh/zasdfgbnm/85/base -> origin/gh/zasdfgbnm/85/base 2022-03-17T19:07:03.9352812Z * [new branch] gh/zasdfgbnm/85/orig -> origin/gh/zasdfgbnm/85/orig 2022-03-17T19:07:03.9354708Z * [new branch] gh/zasdfgbnm/86/base -> origin/gh/zasdfgbnm/86/base 2022-03-17T19:07:03.9356642Z * [new branch] gh/zasdfgbnm/87/base -> origin/gh/zasdfgbnm/87/base 2022-03-17T19:07:03.9358407Z * [new branch] gh/zasdfgbnm/87/orig -> origin/gh/zasdfgbnm/87/orig 2022-03-17T19:07:03.9360192Z * [new branch] gh/zasdfgbnm/88/base -> origin/gh/zasdfgbnm/88/base 2022-03-17T19:07:03.9361539Z * [new branch] gh/zasdfgbnm/88/orig -> origin/gh/zasdfgbnm/88/orig 2022-03-17T19:07:03.9363666Z * [new branch] gh/zasdfgbnm/89/base -> origin/gh/zasdfgbnm/89/base 2022-03-17T19:07:03.9365025Z * [new branch] gh/zasdfgbnm/89/orig -> origin/gh/zasdfgbnm/89/orig 2022-03-17T19:07:03.9366572Z * [new branch] gh/zasdfgbnm/9/base -> origin/gh/zasdfgbnm/9/base 2022-03-17T19:07:03.9367656Z * [new branch] gh/zasdfgbnm/9/head -> origin/gh/zasdfgbnm/9/head 2022-03-17T19:07:03.9368985Z * [new branch] gh/zasdfgbnm/9/orig -> origin/gh/zasdfgbnm/9/orig 2022-03-17T19:07:03.9371312Z * [new branch] gh/zasdfgbnm/90/base -> origin/gh/zasdfgbnm/90/base 2022-03-17T19:07:03.9372450Z * [new branch] gh/zasdfgbnm/90/orig -> origin/gh/zasdfgbnm/90/orig 2022-03-17T19:07:03.9374487Z * [new branch] gh/zasdfgbnm/92/base -> origin/gh/zasdfgbnm/92/base 2022-03-17T19:07:03.9375588Z * [new branch] gh/zasdfgbnm/92/orig -> origin/gh/zasdfgbnm/92/orig 2022-03-17T19:07:03.9377581Z * [new branch] gh/zasdfgbnm/93/base -> origin/gh/zasdfgbnm/93/base 2022-03-17T19:07:03.9378682Z * [new branch] gh/zasdfgbnm/93/orig -> origin/gh/zasdfgbnm/93/orig 2022-03-17T19:07:03.9380752Z * [new branch] gh/zasdfgbnm/94/base -> origin/gh/zasdfgbnm/94/base 2022-03-17T19:07:03.9382943Z * [new branch] gh/zasdfgbnm/94/orig -> origin/gh/zasdfgbnm/94/orig 2022-03-17T19:07:03.9384909Z * [new branch] gh/zasdfgbnm/95/base -> origin/gh/zasdfgbnm/95/base 2022-03-17T19:07:03.9386194Z * [new branch] gh/zasdfgbnm/95/orig -> origin/gh/zasdfgbnm/95/orig 2022-03-17T19:07:03.9388223Z * [new branch] gh/zasdfgbnm/96/base -> origin/gh/zasdfgbnm/96/base 2022-03-17T19:07:03.9389326Z * [new branch] gh/zasdfgbnm/96/orig -> origin/gh/zasdfgbnm/96/orig 2022-03-17T19:07:03.9392052Z * [new branch] gh/zdevito/118/base -> origin/gh/zdevito/118/base 2022-03-17T19:07:03.9393313Z * [new branch] gh/zdevito/118/head -> origin/gh/zdevito/118/head 2022-03-17T19:07:03.9395132Z * [new branch] gh/zdevito/118/orig -> origin/gh/zdevito/118/orig 2022-03-17T19:07:03.9396763Z * [new branch] gh/zdevito/129/base -> origin/gh/zdevito/129/base 2022-03-17T19:07:03.9398082Z * [new branch] gh/zdevito/129/head -> origin/gh/zdevito/129/head 2022-03-17T19:07:03.9399487Z * [new branch] gh/zdevito/129/orig -> origin/gh/zdevito/129/orig 2022-03-17T19:07:03.9401482Z * [new branch] gh/zdevito/143/base -> origin/gh/zdevito/143/base 2022-03-17T19:07:03.9402616Z * [new branch] gh/zdevito/143/head -> origin/gh/zdevito/143/head 2022-03-17T19:07:03.9403931Z * [new branch] gh/zdevito/143/orig -> origin/gh/zdevito/143/orig 2022-03-17T19:07:03.9405984Z * [new branch] gh/zdevito/145/base -> origin/gh/zdevito/145/base 2022-03-17T19:07:03.9407208Z * [new branch] gh/zdevito/145/head -> origin/gh/zdevito/145/head 2022-03-17T19:07:03.9409321Z * [new branch] gh/zdevito/145/orig -> origin/gh/zdevito/145/orig 2022-03-17T19:07:03.9410668Z * [new branch] gh/zdevito/159/base -> origin/gh/zdevito/159/base 2022-03-17T19:07:03.9412018Z * [new branch] gh/zdevito/159/head -> origin/gh/zdevito/159/head 2022-03-17T19:07:03.9413766Z * [new branch] gh/zdevito/159/orig -> origin/gh/zdevito/159/orig 2022-03-17T19:07:03.9415534Z * [new branch] gh/zdevito/160/base -> origin/gh/zdevito/160/base 2022-03-17T19:07:03.9416754Z * [new branch] gh/zdevito/160/head -> origin/gh/zdevito/160/head 2022-03-17T19:07:03.9418635Z * [new branch] gh/zdevito/160/orig -> origin/gh/zdevito/160/orig 2022-03-17T19:07:03.9420396Z * [new branch] gh/zdevito/161/base -> origin/gh/zdevito/161/base 2022-03-17T19:07:03.9421767Z * [new branch] gh/zdevito/161/head -> origin/gh/zdevito/161/head 2022-03-17T19:07:03.9423984Z * [new branch] gh/zdevito/161/orig -> origin/gh/zdevito/161/orig 2022-03-17T19:07:03.9425429Z * [new branch] gh/zdevito/170/base -> origin/gh/zdevito/170/base 2022-03-17T19:07:03.9427125Z * [new branch] gh/zdevito/170/head -> origin/gh/zdevito/170/head 2022-03-17T19:07:03.9428365Z * [new branch] gh/zdevito/170/orig -> origin/gh/zdevito/170/orig 2022-03-17T19:07:03.9430518Z * [new branch] gh/zdevito/171/base -> origin/gh/zdevito/171/base 2022-03-17T19:07:03.9431746Z * [new branch] gh/zdevito/171/head -> origin/gh/zdevito/171/head 2022-03-17T19:07:03.9433130Z * [new branch] gh/zdevito/171/orig -> origin/gh/zdevito/171/orig 2022-03-17T19:07:03.9434737Z * [new branch] gh/zdevito/172/base -> origin/gh/zdevito/172/base 2022-03-17T19:07:03.9436024Z * [new branch] gh/zdevito/172/head -> origin/gh/zdevito/172/head 2022-03-17T19:07:03.9438280Z * [new branch] gh/zdevito/172/orig -> origin/gh/zdevito/172/orig 2022-03-17T19:07:03.9440204Z * [new branch] gh/zdevito/174/base -> origin/gh/zdevito/174/base 2022-03-17T19:07:03.9441423Z * [new branch] gh/zdevito/174/head -> origin/gh/zdevito/174/head 2022-03-17T19:07:03.9442636Z * [new branch] gh/zdevito/174/orig -> origin/gh/zdevito/174/orig 2022-03-17T19:07:03.9444659Z * [new branch] gh/zdevito/181/base -> origin/gh/zdevito/181/base 2022-03-17T19:07:03.9445727Z * [new branch] gh/zdevito/181/head -> origin/gh/zdevito/181/head 2022-03-17T19:07:03.9447073Z * [new branch] gh/zdevito/181/orig -> origin/gh/zdevito/181/orig 2022-03-17T19:07:03.9449252Z * [new branch] gh/zdevito/183/base -> origin/gh/zdevito/183/base 2022-03-17T19:07:03.9450436Z * [new branch] gh/zdevito/183/orig -> origin/gh/zdevito/183/orig 2022-03-17T19:07:03.9452490Z * [new branch] gh/zdevito/184/base -> origin/gh/zdevito/184/base 2022-03-17T19:07:03.9453619Z * [new branch] gh/zdevito/184/head -> origin/gh/zdevito/184/head 2022-03-17T19:07:03.9454943Z * [new branch] gh/zdevito/184/orig -> origin/gh/zdevito/184/orig 2022-03-17T19:07:03.9456877Z * [new branch] gh/zdevito/99/base -> origin/gh/zdevito/99/base 2022-03-17T19:07:03.9458013Z * [new branch] gh/zdevito/99/head -> origin/gh/zdevito/99/head 2022-03-17T19:07:03.9460008Z * [new branch] gh/zdevito/99/orig -> origin/gh/zdevito/99/orig 2022-03-17T19:07:03.9462853Z * [new branch] gh/zhaojuanmao/100/base -> origin/gh/zhaojuanmao/100/base 2022-03-17T19:07:03.9464110Z * [new branch] gh/zhaojuanmao/100/head -> origin/gh/zhaojuanmao/100/head 2022-03-17T19:07:03.9466415Z * [new branch] gh/zhaojuanmao/100/orig -> origin/gh/zhaojuanmao/100/orig 2022-03-17T19:07:03.9468072Z * [new branch] gh/zhaojuanmao/101/base -> origin/gh/zhaojuanmao/101/base 2022-03-17T19:07:03.9469375Z * [new branch] gh/zhaojuanmao/101/head -> origin/gh/zhaojuanmao/101/head 2022-03-17T19:07:03.9471312Z * [new branch] gh/zhaojuanmao/101/orig -> origin/gh/zhaojuanmao/101/orig 2022-03-17T19:07:03.9473243Z * [new branch] gh/zhaojuanmao/5/head -> origin/gh/zhaojuanmao/5/head 2022-03-17T19:07:03.9474237Z * [new branch] gh/zhaojuanmao/5/orig -> origin/gh/zhaojuanmao/5/orig 2022-03-17T19:07:03.9476241Z * [new branch] gh/zhaojuanmao/55/base -> origin/gh/zhaojuanmao/55/base 2022-03-17T19:07:03.9477506Z * [new branch] gh/zhaojuanmao/55/head -> origin/gh/zhaojuanmao/55/head 2022-03-17T19:07:03.9479127Z * [new branch] gh/zhaojuanmao/55/orig -> origin/gh/zhaojuanmao/55/orig 2022-03-17T19:07:03.9481144Z * [new branch] gh/zhaojuanmao/61/base -> origin/gh/zhaojuanmao/61/base 2022-03-17T19:07:03.9482301Z * [new branch] gh/zhaojuanmao/61/head -> origin/gh/zhaojuanmao/61/head 2022-03-17T19:07:03.9483667Z * [new branch] gh/zhaojuanmao/61/orig -> origin/gh/zhaojuanmao/61/orig 2022-03-17T19:07:03.9485667Z * [new branch] gh/zhaojuanmao/68/base -> origin/gh/zhaojuanmao/68/base 2022-03-17T19:07:03.9486938Z * [new branch] gh/zhaojuanmao/68/head -> origin/gh/zhaojuanmao/68/head 2022-03-17T19:07:03.9488324Z * [new branch] gh/zhaojuanmao/68/orig -> origin/gh/zhaojuanmao/68/orig 2022-03-17T19:07:03.9490557Z * [new branch] gh/zhaojuanmao/69/base -> origin/gh/zhaojuanmao/69/base 2022-03-17T19:07:03.9491786Z * [new branch] gh/zhaojuanmao/69/head -> origin/gh/zhaojuanmao/69/head 2022-03-17T19:07:03.9493594Z * [new branch] gh/zhaojuanmao/69/orig -> origin/gh/zhaojuanmao/69/orig 2022-03-17T19:07:03.9495447Z * [new branch] gh/zhaojuanmao/74/base -> origin/gh/zhaojuanmao/74/base 2022-03-17T19:07:03.9496595Z * [new branch] gh/zhaojuanmao/74/head -> origin/gh/zhaojuanmao/74/head 2022-03-17T19:07:03.9497959Z * [new branch] gh/zhaojuanmao/74/orig -> origin/gh/zhaojuanmao/74/orig 2022-03-17T19:07:03.9500638Z * [new branch] gh/zhaojuanmao/80/base -> origin/gh/zhaojuanmao/80/base 2022-03-17T19:07:03.9502101Z * [new branch] gh/zhaojuanmao/80/head -> origin/gh/zhaojuanmao/80/head 2022-03-17T19:07:03.9504252Z * [new branch] gh/zhaojuanmao/80/orig -> origin/gh/zhaojuanmao/80/orig 2022-03-17T19:07:03.9506192Z * [new branch] gh/zhaojuanmao/81/base -> origin/gh/zhaojuanmao/81/base 2022-03-17T19:07:03.9507506Z * [new branch] gh/zhaojuanmao/81/head -> origin/gh/zhaojuanmao/81/head 2022-03-17T19:07:03.9509224Z * [new branch] gh/zhaojuanmao/81/orig -> origin/gh/zhaojuanmao/81/orig 2022-03-17T19:07:03.9510559Z * [new branch] gh/zhaojuanmao/82/base -> origin/gh/zhaojuanmao/82/base 2022-03-17T19:07:03.9512472Z * [new branch] gh/zhaojuanmao/82/head -> origin/gh/zhaojuanmao/82/head 2022-03-17T19:07:03.9513634Z * [new branch] gh/zhaojuanmao/82/orig -> origin/gh/zhaojuanmao/82/orig 2022-03-17T19:07:03.9515660Z * [new branch] gh/zhaojuanmao/89/base -> origin/gh/zhaojuanmao/89/base 2022-03-17T19:07:03.9516805Z * [new branch] gh/zhaojuanmao/89/head -> origin/gh/zhaojuanmao/89/head 2022-03-17T19:07:03.9518793Z * [new branch] gh/zhaojuanmao/98/base -> origin/gh/zhaojuanmao/98/base 2022-03-17T19:07:03.9520181Z * [new branch] gh/zhaojuanmao/98/head -> origin/gh/zhaojuanmao/98/head 2022-03-17T19:07:03.9521901Z * [new branch] gh/zhaojuanmao/98/orig -> origin/gh/zhaojuanmao/98/orig 2022-03-17T19:07:03.9523841Z * [new branch] gh/zhaojuanmao/99/base -> origin/gh/zhaojuanmao/99/base 2022-03-17T19:07:03.9524948Z * [new branch] gh/zhaojuanmao/99/head -> origin/gh/zhaojuanmao/99/head 2022-03-17T19:07:03.9526412Z * [new branch] gh/zhaojuanmao/99/orig -> origin/gh/zhaojuanmao/99/orig 2022-03-17T19:07:03.9528656Z * [new branch] gh/zheng-xq/1/base -> origin/gh/zheng-xq/1/base 2022-03-17T19:07:03.9530131Z * [new branch] gh/zheng-xq/1/head -> origin/gh/zheng-xq/1/head 2022-03-17T19:07:03.9532074Z * [new branch] gh/zheng-xq/1/orig -> origin/gh/zheng-xq/1/orig 2022-03-17T19:07:03.9533265Z * [new branch] gh/zheng-xq/2/base -> origin/gh/zheng-xq/2/base 2022-03-17T19:07:03.9534878Z * [new branch] gh/zheng-xq/2/head -> origin/gh/zheng-xq/2/head 2022-03-17T19:07:03.9536155Z * [new branch] gh/zheng-xq/2/orig -> origin/gh/zheng-xq/2/orig 2022-03-17T19:07:03.9538221Z * [new branch] gh/zheng-xq/4/base -> origin/gh/zheng-xq/4/base 2022-03-17T19:07:03.9539292Z * [new branch] gh/zheng-xq/4/head -> origin/gh/zheng-xq/4/head 2022-03-17T19:07:03.9540771Z * [new branch] gh/zheng-xq/4/orig -> origin/gh/zheng-xq/4/orig 2022-03-17T19:07:03.9543640Z * [new branch] gh/zhxchen17/13/base -> origin/gh/zhxchen17/13/base 2022-03-17T19:07:03.9544872Z * [new branch] gh/zhxchen17/13/head -> origin/gh/zhxchen17/13/head 2022-03-17T19:07:03.9546666Z * [new branch] gh/zhxchen17/13/orig -> origin/gh/zhxchen17/13/orig 2022-03-17T19:07:03.9549075Z * [new branch] gh/zhxchen17/14/base -> origin/gh/zhxchen17/14/base 2022-03-17T19:07:03.9550327Z * [new branch] gh/zhxchen17/14/head -> origin/gh/zhxchen17/14/head 2022-03-17T19:07:03.9552270Z * [new branch] gh/zhxchen17/15/base -> origin/gh/zhxchen17/15/base 2022-03-17T19:07:03.9553281Z * [new branch] gh/zhxchen17/15/head -> origin/gh/zhxchen17/15/head 2022-03-17T19:07:03.9554786Z * [new branch] gh/zhxchen17/16/base -> origin/gh/zhxchen17/16/base 2022-03-17T19:07:03.9556015Z * [new branch] gh/zhxchen17/16/head -> origin/gh/zhxchen17/16/head 2022-03-17T19:07:03.9557806Z * [new branch] gh/zhxchen17/17/base -> origin/gh/zhxchen17/17/base 2022-03-17T19:07:03.9558778Z * [new branch] gh/zhxchen17/17/head -> origin/gh/zhxchen17/17/head 2022-03-17T19:07:03.9560475Z * [new branch] gh/zhxchen17/18/base -> origin/gh/zhxchen17/18/base 2022-03-17T19:07:03.9561546Z * [new branch] gh/zhxchen17/18/head -> origin/gh/zhxchen17/18/head 2022-03-17T19:07:03.9563234Z * [new branch] gh/zhxchen17/19/base -> origin/gh/zhxchen17/19/base 2022-03-17T19:07:03.9564363Z * [new branch] gh/zhxchen17/19/head -> origin/gh/zhxchen17/19/head 2022-03-17T19:07:03.9565938Z * [new branch] gh/zhxchen17/20/base -> origin/gh/zhxchen17/20/base 2022-03-17T19:07:03.9567860Z * [new branch] gh/zhxchen17/20/head -> origin/gh/zhxchen17/20/head 2022-03-17T19:07:03.9568946Z * [new branch] gh/zhxchen17/21/base -> origin/gh/zhxchen17/21/base 2022-03-17T19:07:03.9570157Z * [new branch] gh/zhxchen17/21/head -> origin/gh/zhxchen17/21/head 2022-03-17T19:07:03.9572212Z * [new branch] gh/zhxchen17/25/base -> origin/gh/zhxchen17/25/base 2022-03-17T19:07:03.9573262Z * [new branch] gh/zhxchen17/25/head -> origin/gh/zhxchen17/25/head 2022-03-17T19:07:03.9574673Z * [new branch] gh/zhxchen17/25/orig -> origin/gh/zhxchen17/25/orig 2022-03-17T19:07:03.9576641Z * [new branch] gh/zhxchen17/26/base -> origin/gh/zhxchen17/26/base 2022-03-17T19:07:03.9577815Z * [new branch] gh/zhxchen17/26/orig -> origin/gh/zhxchen17/26/orig 2022-03-17T19:07:03.9579897Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2022-03-17T19:07:03.9581133Z * [new branch] gh/zhxchen17/35/orig -> origin/gh/zhxchen17/35/orig 2022-03-17T19:07:03.9583755Z * [new branch] gh/zhxchen17/36/base -> origin/gh/zhxchen17/36/base 2022-03-17T19:07:03.9585832Z * [new branch] gh/zhxchen17/36/orig -> origin/gh/zhxchen17/36/orig 2022-03-17T19:07:03.9587763Z * [new branch] gh/zhxchen17/37/base -> origin/gh/zhxchen17/37/base 2022-03-17T19:07:03.9588775Z * [new branch] gh/zhxchen17/37/orig -> origin/gh/zhxchen17/37/orig 2022-03-17T19:07:03.9591463Z * [new branch] gh/zhxchen17/38/base -> origin/gh/zhxchen17/38/base 2022-03-17T19:07:03.9592628Z * [new branch] gh/zhxchen17/38/orig -> origin/gh/zhxchen17/38/orig 2022-03-17T19:07:03.9595111Z * [new branch] gh/zhxchen17/39/base -> origin/gh/zhxchen17/39/base 2022-03-17T19:07:03.9596220Z * [new branch] gh/zhxchen17/39/orig -> origin/gh/zhxchen17/39/orig 2022-03-17T19:07:03.9598263Z * [new branch] gh/zhxchen17/4/base -> origin/gh/zhxchen17/4/base 2022-03-17T19:07:03.9599494Z * [new branch] gh/zhxchen17/4/orig -> origin/gh/zhxchen17/4/orig 2022-03-17T19:07:03.9601440Z * [new branch] gh/zhxchen17/40/base -> origin/gh/zhxchen17/40/base 2022-03-17T19:07:03.9602502Z * [new branch] gh/zhxchen17/40/orig -> origin/gh/zhxchen17/40/orig 2022-03-17T19:07:03.9604490Z * [new branch] gh/zhxchen17/41/base -> origin/gh/zhxchen17/41/base 2022-03-17T19:07:03.9605841Z * [new branch] gh/zhxchen17/41/orig -> origin/gh/zhxchen17/41/orig 2022-03-17T19:07:03.9607949Z * [new branch] gh/zhxchen17/42/base -> origin/gh/zhxchen17/42/base 2022-03-17T19:07:03.9609216Z * [new branch] gh/zhxchen17/42/orig -> origin/gh/zhxchen17/42/orig 2022-03-17T19:07:03.9611280Z * [new branch] gh/zhxchen17/69/base -> origin/gh/zhxchen17/69/base 2022-03-17T19:07:03.9612410Z * [new branch] gh/zhxchen17/69/head -> origin/gh/zhxchen17/69/head 2022-03-17T19:07:03.9613957Z * [new branch] gh/zhxchen17/69/orig -> origin/gh/zhxchen17/69/orig 2022-03-17T19:07:03.9615949Z * [new branch] gh/zhxchen17/7/base -> origin/gh/zhxchen17/7/base 2022-03-17T19:07:03.9617167Z * [new branch] gh/zhxchen17/7/head -> origin/gh/zhxchen17/7/head 2022-03-17T19:07:03.9618876Z * [new branch] gh/zhxchen17/7/orig -> origin/gh/zhxchen17/7/orig 2022-03-17T19:07:03.9620817Z * [new branch] gh/zhxchen17/79/base -> origin/gh/zhxchen17/79/base 2022-03-17T19:07:03.9622180Z * [new branch] gh/zhxchen17/79/head -> origin/gh/zhxchen17/79/head 2022-03-17T19:07:03.9624245Z * [new branch] gh/zhxchen17/79/orig -> origin/gh/zhxchen17/79/orig 2022-03-17T19:07:03.9626442Z * [new branch] gh/zmvictor/1/base -> origin/gh/zmvictor/1/base 2022-03-17T19:07:03.9627611Z * [new branch] gh/zmvictor/1/head -> origin/gh/zmvictor/1/head 2022-03-17T19:07:03.9630003Z * [new branch] gh/zou3519/117/base -> origin/gh/zou3519/117/base 2022-03-17T19:07:03.9631261Z * [new branch] gh/zou3519/117/head -> origin/gh/zou3519/117/head 2022-03-17T19:07:03.9633040Z * [new branch] gh/zou3519/117/orig -> origin/gh/zou3519/117/orig 2022-03-17T19:07:03.9634444Z * [new branch] gh/zou3519/118/base -> origin/gh/zou3519/118/base 2022-03-17T19:07:03.9635926Z * [new branch] gh/zou3519/118/head -> origin/gh/zou3519/118/head 2022-03-17T19:07:03.9637613Z * [new branch] gh/zou3519/118/orig -> origin/gh/zou3519/118/orig 2022-03-17T19:07:03.9638965Z * [new branch] gh/zou3519/119/base -> origin/gh/zou3519/119/base 2022-03-17T19:07:03.9640649Z * [new branch] gh/zou3519/119/head -> origin/gh/zou3519/119/head 2022-03-17T19:07:03.9641914Z * [new branch] gh/zou3519/119/orig -> origin/gh/zou3519/119/orig 2022-03-17T19:07:03.9643878Z * [new branch] gh/zou3519/122/base -> origin/gh/zou3519/122/base 2022-03-17T19:07:03.9645122Z * [new branch] gh/zou3519/122/head -> origin/gh/zou3519/122/head 2022-03-17T19:07:03.9646591Z * [new branch] gh/zou3519/122/orig -> origin/gh/zou3519/122/orig 2022-03-17T19:07:03.9648629Z * [new branch] gh/zou3519/144/base -> origin/gh/zou3519/144/base 2022-03-17T19:07:03.9649817Z * [new branch] gh/zou3519/144/head -> origin/gh/zou3519/144/head 2022-03-17T19:07:03.9651642Z * [new branch] gh/zou3519/144/orig -> origin/gh/zou3519/144/orig 2022-03-17T19:07:03.9653605Z * [new branch] gh/zou3519/155/head -> origin/gh/zou3519/155/head 2022-03-17T19:07:03.9654826Z * [new branch] gh/zou3519/155/orig -> origin/gh/zou3519/155/orig 2022-03-17T19:07:03.9656907Z * [new branch] gh/zou3519/162/head -> origin/gh/zou3519/162/head 2022-03-17T19:07:03.9658080Z * [new branch] gh/zou3519/162/orig -> origin/gh/zou3519/162/orig 2022-03-17T19:07:03.9660076Z * [new branch] gh/zou3519/207/base -> origin/gh/zou3519/207/base 2022-03-17T19:07:03.9661401Z * [new branch] gh/zou3519/207/head -> origin/gh/zou3519/207/head 2022-03-17T19:07:03.9664160Z * [new branch] gh/zou3519/207/orig -> origin/gh/zou3519/207/orig 2022-03-17T19:07:03.9665738Z * [new branch] gh/zou3519/208/base -> origin/gh/zou3519/208/base 2022-03-17T19:07:03.9667686Z * [new branch] gh/zou3519/208/head -> origin/gh/zou3519/208/head 2022-03-17T19:07:03.9669751Z * [new branch] gh/zou3519/208/orig -> origin/gh/zou3519/208/orig 2022-03-17T19:07:03.9671663Z * [new branch] gh/zou3519/219/base -> origin/gh/zou3519/219/base 2022-03-17T19:07:03.9673077Z * [new branch] gh/zou3519/219/head -> origin/gh/zou3519/219/head 2022-03-17T19:07:03.9674985Z * [new branch] gh/zou3519/219/orig -> origin/gh/zou3519/219/orig 2022-03-17T19:07:03.9677310Z * [new branch] gh/zou3519/224/base -> origin/gh/zou3519/224/base 2022-03-17T19:07:03.9678536Z * [new branch] gh/zou3519/224/head -> origin/gh/zou3519/224/head 2022-03-17T19:07:03.9679875Z * [new branch] gh/zou3519/224/orig -> origin/gh/zou3519/224/orig 2022-03-17T19:07:03.9682001Z * [new branch] gh/zou3519/251/base -> origin/gh/zou3519/251/base 2022-03-17T19:07:03.9683256Z * [new branch] gh/zou3519/251/head -> origin/gh/zou3519/251/head 2022-03-17T19:07:03.9685079Z * [new branch] gh/zou3519/251/orig -> origin/gh/zou3519/251/orig 2022-03-17T19:07:03.9686546Z * [new branch] gh/zou3519/252/base -> origin/gh/zou3519/252/base 2022-03-17T19:07:03.9688544Z * [new branch] gh/zou3519/252/head -> origin/gh/zou3519/252/head 2022-03-17T19:07:03.9690202Z * [new branch] gh/zou3519/252/orig -> origin/gh/zou3519/252/orig 2022-03-17T19:07:03.9691854Z * [new branch] gh/zou3519/277/base -> origin/gh/zou3519/277/base 2022-03-17T19:07:03.9693202Z * [new branch] gh/zou3519/277/head -> origin/gh/zou3519/277/head 2022-03-17T19:07:03.9694582Z * [new branch] gh/zou3519/277/orig -> origin/gh/zou3519/277/orig 2022-03-17T19:07:03.9697137Z * [new branch] gh/zou3519/284/base -> origin/gh/zou3519/284/base 2022-03-17T19:07:03.9698291Z * [new branch] gh/zou3519/284/head -> origin/gh/zou3519/284/head 2022-03-17T19:07:03.9699766Z * [new branch] gh/zou3519/284/orig -> origin/gh/zou3519/284/orig 2022-03-17T19:07:03.9702049Z * [new branch] gh/zou3519/293/base -> origin/gh/zou3519/293/base 2022-03-17T19:07:03.9703728Z * [new branch] gh/zou3519/293/head -> origin/gh/zou3519/293/head 2022-03-17T19:07:03.9705022Z * [new branch] gh/zou3519/293/orig -> origin/gh/zou3519/293/orig 2022-03-17T19:07:03.9706992Z * [new branch] gh/zou3519/298/base -> origin/gh/zou3519/298/base 2022-03-17T19:07:03.9708814Z * [new branch] gh/zou3519/298/head -> origin/gh/zou3519/298/head 2022-03-17T19:07:03.9709771Z * [new branch] gh/zou3519/298/orig -> origin/gh/zou3519/298/orig 2022-03-17T19:07:03.9711548Z * [new branch] gh/zou3519/308/base -> origin/gh/zou3519/308/base 2022-03-17T19:07:03.9712881Z * [new branch] gh/zou3519/308/head -> origin/gh/zou3519/308/head 2022-03-17T19:07:03.9714868Z * [new branch] gh/zou3519/308/orig -> origin/gh/zou3519/308/orig 2022-03-17T19:07:03.9716799Z * [new branch] gh/zou3519/311/base -> origin/gh/zou3519/311/base 2022-03-17T19:07:03.9718161Z * [new branch] gh/zou3519/311/head -> origin/gh/zou3519/311/head 2022-03-17T19:07:03.9720110Z * [new branch] gh/zou3519/311/orig -> origin/gh/zou3519/311/orig 2022-03-17T19:07:03.9721246Z * [new branch] gh/zou3519/312/base -> origin/gh/zou3519/312/base 2022-03-17T19:07:03.9723174Z * [new branch] gh/zou3519/312/head -> origin/gh/zou3519/312/head 2022-03-17T19:07:03.9724457Z * [new branch] gh/zou3519/312/orig -> origin/gh/zou3519/312/orig 2022-03-17T19:07:03.9726333Z * [new branch] gh/zou3519/313/base -> origin/gh/zou3519/313/base 2022-03-17T19:07:03.9727582Z * [new branch] gh/zou3519/313/head -> origin/gh/zou3519/313/head 2022-03-17T19:07:03.9729590Z * [new branch] gh/zou3519/313/orig -> origin/gh/zou3519/313/orig 2022-03-17T19:07:03.9731490Z * [new branch] gh/zou3519/339/base -> origin/gh/zou3519/339/base 2022-03-17T19:07:03.9732785Z * [new branch] gh/zou3519/339/head -> origin/gh/zou3519/339/head 2022-03-17T19:07:03.9734487Z * [new branch] gh/zou3519/339/orig -> origin/gh/zou3519/339/orig 2022-03-17T19:07:03.9736406Z * [new branch] gh/zou3519/348/base -> origin/gh/zou3519/348/base 2022-03-17T19:07:03.9737545Z * [new branch] gh/zou3519/348/head -> origin/gh/zou3519/348/head 2022-03-17T19:07:03.9739211Z * [new branch] gh/zou3519/348/orig -> origin/gh/zou3519/348/orig 2022-03-17T19:07:03.9741250Z * [new branch] gh/zou3519/351/base -> origin/gh/zou3519/351/base 2022-03-17T19:07:03.9742619Z * [new branch] gh/zou3519/351/head -> origin/gh/zou3519/351/head 2022-03-17T19:07:03.9744015Z * [new branch] gh/zou3519/351/orig -> origin/gh/zou3519/351/orig 2022-03-17T19:07:03.9746760Z * [new branch] gh/zou3519/382/base -> origin/gh/zou3519/382/base 2022-03-17T19:07:03.9747943Z * [new branch] gh/zou3519/382/head -> origin/gh/zou3519/382/head 2022-03-17T19:07:03.9749321Z * [new branch] gh/zou3519/382/orig -> origin/gh/zou3519/382/orig 2022-03-17T19:07:03.9751524Z * [new branch] gh/zou3519/387/base -> origin/gh/zou3519/387/base 2022-03-17T19:07:03.9752739Z * [new branch] gh/zou3519/387/head -> origin/gh/zou3519/387/head 2022-03-17T19:07:03.9754137Z * [new branch] gh/zou3519/387/orig -> origin/gh/zou3519/387/orig 2022-03-17T19:07:03.9756141Z * [new branch] gh/zou3519/400/base -> origin/gh/zou3519/400/base 2022-03-17T19:07:03.9757397Z * [new branch] gh/zou3519/400/head -> origin/gh/zou3519/400/head 2022-03-17T19:07:03.9759326Z * [new branch] gh/zou3519/400/orig -> origin/gh/zou3519/400/orig 2022-03-17T19:07:03.9761111Z * [new branch] gh/zou3519/403/base -> origin/gh/zou3519/403/base 2022-03-17T19:07:03.9762203Z * [new branch] gh/zou3519/403/head -> origin/gh/zou3519/403/head 2022-03-17T19:07:03.9763542Z * [new branch] gh/zou3519/403/orig -> origin/gh/zou3519/403/orig 2022-03-17T19:07:03.9765570Z * [new branch] gh/zou3519/404/base -> origin/gh/zou3519/404/base 2022-03-17T19:07:03.9766798Z * [new branch] gh/zou3519/404/head -> origin/gh/zou3519/404/head 2022-03-17T19:07:03.9768149Z * [new branch] gh/zou3519/404/orig -> origin/gh/zou3519/404/orig 2022-03-17T19:07:03.9770406Z * [new branch] gh/zou3519/408/base -> origin/gh/zou3519/408/base 2022-03-17T19:07:03.9771460Z * [new branch] gh/zou3519/408/head -> origin/gh/zou3519/408/head 2022-03-17T19:07:03.9772793Z * [new branch] gh/zou3519/408/orig -> origin/gh/zou3519/408/orig 2022-03-17T19:07:03.9775497Z * [new branch] gh/zou3519/411/base -> origin/gh/zou3519/411/base 2022-03-17T19:07:03.9776693Z * [new branch] gh/zou3519/411/head -> origin/gh/zou3519/411/head 2022-03-17T19:07:03.9778042Z * [new branch] gh/zou3519/411/orig -> origin/gh/zou3519/411/orig 2022-03-17T19:07:03.9780228Z * [new branch] gh/zou3519/412/base -> origin/gh/zou3519/412/base 2022-03-17T19:07:03.9781701Z * [new branch] gh/zou3519/412/head -> origin/gh/zou3519/412/head 2022-03-17T19:07:03.9783899Z * [new branch] gh/zou3519/412/orig -> origin/gh/zou3519/412/orig 2022-03-17T19:07:03.9785744Z * [new branch] gh/zou3519/413/base -> origin/gh/zou3519/413/base 2022-03-17T19:07:03.9786911Z * [new branch] gh/zou3519/413/head -> origin/gh/zou3519/413/head 2022-03-17T19:07:03.9788371Z * [new branch] gh/zou3519/413/orig -> origin/gh/zou3519/413/orig 2022-03-17T19:07:03.9790392Z * [new branch] gh/zou3519/414/base -> origin/gh/zou3519/414/base 2022-03-17T19:07:03.9791843Z * [new branch] gh/zou3519/414/head -> origin/gh/zou3519/414/head 2022-03-17T19:07:03.9793341Z * [new branch] gh/zou3519/414/orig -> origin/gh/zou3519/414/orig 2022-03-17T19:07:03.9795396Z * [new branch] gh/zou3519/415/base -> origin/gh/zou3519/415/base 2022-03-17T19:07:03.9796459Z * [new branch] gh/zou3519/415/head -> origin/gh/zou3519/415/head 2022-03-17T19:07:03.9798375Z * [new branch] gh/zou3519/415/orig -> origin/gh/zou3519/415/orig 2022-03-17T19:07:03.9799715Z * [new branch] gh/zou3519/87/base -> origin/gh/zou3519/87/base 2022-03-17T19:07:03.9801683Z * [new branch] gh/zou3519/87/head -> origin/gh/zou3519/87/head 2022-03-17T19:07:03.9802953Z * [new branch] gh/zou3519/87/orig -> origin/gh/zou3519/87/orig 2022-03-17T19:07:03.9805257Z * [new branch] gh/zzzwen/1/base -> origin/gh/zzzwen/1/base 2022-03-17T19:07:03.9806396Z * [new branch] gh/zzzwen/1/orig -> origin/gh/zzzwen/1/orig 2022-03-17T19:07:03.9808442Z * [new branch] gh/zzzwen/2/base -> origin/gh/zzzwen/2/base 2022-03-17T19:07:03.9809573Z * [new branch] gh/zzzwen/2/head -> origin/gh/zzzwen/2/head 2022-03-17T19:07:03.9811506Z * [new branch] gh/zzzwen/2/orig -> origin/gh/zzzwen/2/orig 2022-03-17T19:07:03.9812874Z * [new branch] git_submodule_docs -> origin/git_submodule_docs 2022-03-17T19:07:03.9814826Z * [new branch] gj/Ivan -> origin/gj/Ivan 2022-03-17T19:07:03.9816198Z * [new branch] gradle_bump -> origin/gradle_bump 2022-03-17T19:07:03.9818119Z * [new branch] hagiopol -> origin/hagiopol 2022-03-17T19:07:03.9819451Z * [new branch] hcz_debug -> origin/hcz_debug 2022-03-17T19:07:03.9822096Z * [new branch] heitorschueroff/einsum/optimize -> origin/heitorschueroff/einsum/optimize 2022-03-17T19:07:03.9823410Z * [new branch] heitorschueroff/minmax -> origin/heitorschueroff/minmax 2022-03-17T19:07:03.9825381Z * [new branch] heitorschueroff/opinfo/fails -> origin/heitorschueroff/opinfo/fails 2022-03-17T19:07:03.9826457Z * [new branch] heitorschueroff/opinfo/reductions -> origin/heitorschueroff/opinfo/reductions 2022-03-17T19:07:03.9828410Z * [new branch] heitorschueroff/quantile/interpolation -> origin/heitorschueroff/quantile/interpolation 2022-03-17T19:07:03.9829482Z * [new branch] helper -> origin/helper 2022-03-17T19:07:03.9831523Z * [new branch] hide_internal_API -> origin/hide_internal_API 2022-03-17T19:07:03.9832765Z * [new branch] hide_set_enabled -> origin/hide_set_enabled 2022-03-17T19:07:03.9834599Z * [new branch] histc_docs_fix -> origin/histc_docs_fix 2022-03-17T19:07:03.9835706Z * [new branch] histc_opinfo -> origin/histc_opinfo 2022-03-17T19:07:03.9837187Z * [new branch] histc_perf_fix -> origin/histc_perf_fix 2022-03-17T19:07:03.9839179Z * [new branch] histogramdd -> origin/histogramdd 2022-03-17T19:07:03.9840549Z * [new branch] histogramdd_rsts -> origin/histogramdd_rsts 2022-03-17T19:07:03.9842204Z * [new branch] holly1238-patch-1 -> origin/holly1238-patch-1 2022-03-17T19:07:03.9843656Z * [new branch] hot-fix-librosa -> origin/hot-fix-librosa 2022-03-17T19:07:03.9845741Z * [new branch] hotfix_packaging_version -> origin/hotfix_packaging_version 2022-03-17T19:07:03.9846978Z * [new branch] hub_improvements -> origin/hub_improvements 2022-03-17T19:07:03.9848855Z * [new branch] hub_security -> origin/hub_security 2022-03-17T19:07:03.9850016Z * [new branch] hufsb -> origin/hufsb 2022-03-17T19:07:03.9851970Z * [new branch] igorsyl/1 -> origin/igorsyl/1 2022-03-17T19:07:03.9853334Z * [new branch] ik-test -> origin/ik-test 2022-03-17T19:07:03.9855264Z * [new branch] ik_android_byte_tensors_tests -> origin/ik_android_byte_tensors_tests 2022-03-17T19:07:03.9857385Z * [new branch] ik_android_generic_dict_issue -> origin/ik_android_generic_dict_issue 2022-03-17T19:07:03.9858741Z * [new branch] ik_android_gradle_bintray_publishing -> origin/ik_android_gradle_bintray_publishing 2022-03-17T19:07:03.9860127Z * [new branch] ik_android_vision_libyuv -> origin/ik_android_vision_libyuv 2022-03-17T19:07:03.9863183Z * [new branch] ik_lapack_qml -> origin/ik_lapack_qml 2022-03-17T19:07:03.9864596Z * [new branch] ik_vulkan_ci -> origin/ik_vulkan_ci 2022-03-17T19:07:03.9866530Z * [new branch] ik_wip_android_stdouterr_to_logcat -> origin/ik_wip_android_stdouterr_to_logcat 2022-03-17T19:07:03.9867736Z * [new branch] illegal_memory_access -> origin/illegal_memory_access 2022-03-17T19:07:03.9869243Z * [new branch] independent-constraint -> origin/independent-constraint 2022-03-17T19:07:03.9870907Z * [new branch] independent-transform -> origin/independent-transform 2022-03-17T19:07:03.9872037Z * [new branch] infer_types -> origin/infer_types 2022-03-17T19:07:03.9873830Z * [new branch] infer_types_methods -> origin/infer_types_methods 2022-03-17T19:07:03.9875126Z * [new branch] informative-dist-support-error -> origin/informative-dist-support-error 2022-03-17T19:07:03.9876746Z * [new branch] inplaceorview-test-ci -> origin/inplaceorview-test-ci 2022-03-17T19:07:03.9878516Z * [new branch] iotamudelta/update_rocm3.1 -> origin/iotamudelta/update_rocm3.1 2022-03-17T19:07:03.9879640Z * [new branch] isin-feature -> origin/isin-feature 2022-03-17T19:07:03.9881346Z * [new branch] issue#58739 -> origin/issue#58739 2022-03-17T19:07:03.9882943Z * [new branch] issue6009 -> origin/issue6009 2022-03-17T19:07:03.9885272Z * [new branch] ivan/add-symbol-qr -> origin/ivan/add-symbol-qr 2022-03-17T19:07:03.9886662Z * [new branch] jamesr66a-patch-1 -> origin/jamesr66a-patch-1 2022-03-17T19:07:03.9888252Z * [new branch] jamesr66a-patch-2 -> origin/jamesr66a-patch-2 2022-03-17T19:07:03.9889516Z * [new branch] jenkins_and_tools_update_master_to_main -> origin/jenkins_and_tools_update_master_to_main 2022-03-17T19:07:03.9891373Z * [new branch] jerryzh168-patch-6 -> origin/jerryzh168-patch-6 2022-03-17T19:07:03.9893169Z * [new branch] jit-language-reference-types-ii -> origin/jit-language-reference-types-ii 2022-03-17T19:07:03.9895086Z * [new branch] jit-language-spec -> origin/jit-language-spec 2022-03-17T19:07:03.9896310Z * [new branch] jit3.9.1 -> origin/jit3.9.1 2022-03-17T19:07:03.9898078Z * [new branch] jit_future_overview_fix -> origin/jit_future_overview_fix 2022-03-17T19:07:03.9899477Z * [new branch] jiterator_documentation -> origin/jiterator_documentation 2022-03-17T19:07:03.9900894Z * [new branch] jlin27-docs-toc -> origin/jlin27-docs-toc 2022-03-17T19:07:03.9903283Z * [new branch] jlin27-sphinx-tag -> origin/jlin27-sphinx-tag 2022-03-17T19:07:03.9905029Z * [new branch] jlin27-sphinx-tag-2 -> origin/jlin27-sphinx-tag-2 2022-03-17T19:07:03.9906405Z * [new branch] jspisak-patch-1 -> origin/jspisak-patch-1 2022-03-17T19:07:03.9908147Z * [new branch] jspisak-patch-2 -> origin/jspisak-patch-2 2022-03-17T19:07:03.9909778Z * [new branch] jspisak-patch-3 -> origin/jspisak-patch-3 2022-03-17T19:07:03.9911498Z * [new branch] kerryz/test-ci -> origin/kerryz/test-ci 2022-03-17T19:07:03.9912844Z * [new branch] kill_mixin -> origin/kill_mixin 2022-03-17T19:07:03.9914436Z * [new branch] kineto_exp -> origin/kineto_exp 2022-03-17T19:07:03.9915615Z * [new branch] kineto_metadata -> origin/kineto_metadata 2022-03-17T19:07:03.9917821Z * [new branch] kiuk/cp-66182 -> origin/kiuk/cp-66182 2022-03-17T19:07:03.9920215Z * [new branch] krovatkin/bench_timings -> origin/krovatkin/bench_timings 2022-03-17T19:07:03.9921814Z * [new branch] krovatkin/conv2d_out -> origin/krovatkin/conv2d_out 2022-03-17T19:07:03.9922734Z * [new branch] krovatkin/cse_changed -> origin/krovatkin/cse_changed 2022-03-17T19:07:03.9924441Z * [new branch] krovatkin/dummy_torchbench -> origin/krovatkin/dummy_torchbench 2022-03-17T19:07:03.9925671Z * [new branch] krovatkin/gamrix_25 -> origin/krovatkin/gamrix_25 2022-03-17T19:07:03.9927049Z * [new branch] krovatkin/improve_error -> origin/krovatkin/improve_error 2022-03-17T19:07:03.9928206Z * [new branch] krovatkin/inline_threshold -> origin/krovatkin/inline_threshold 2022-03-17T19:07:03.9929598Z * [new branch] krovatkin/inline_threshold2 -> origin/krovatkin/inline_threshold2 2022-03-17T19:07:03.9930968Z * [new branch] krovatkin/kevin_assert -> origin/krovatkin/kevin_assert 2022-03-17T19:07:03.9932337Z * [new branch] krovatkin/opt_for_infer -> origin/krovatkin/opt_for_infer 2022-03-17T19:07:03.9933507Z * [new branch] krovatkin/opt_for_infr -> origin/krovatkin/opt_for_infr 2022-03-17T19:07:03.9935154Z * [new branch] krovatkin/opt_for_infr2 -> origin/krovatkin/opt_for_infr2 2022-03-17T19:07:03.9936085Z * [new branch] krovatkin/reformat -> origin/krovatkin/reformat 2022-03-17T19:07:03.9937732Z * [new branch] krovatkin/squeeze -> origin/krovatkin/squeeze 2022-03-17T19:07:03.9939456Z * [new branch] krovatkin/squeeze_master -> origin/krovatkin/squeeze_master 2022-03-17T19:07:04.0026750Z * [new branch] krovatkin/sub_lazy_bench -> origin/krovatkin/sub_lazy_bench 2022-03-17T19:07:04.0041998Z * [new branch] krovatkin/test_legacy_fuser -> origin/krovatkin/test_legacy_fuser 2022-03-17T19:07:04.0043536Z * [new branch] krovatkin/unsqueeze_ltc -> origin/krovatkin/unsqueeze_ltc 2022-03-17T19:07:04.0045569Z * [new branch] ktse/codegen_exercises -> origin/ktse/codegen_exercises 2022-03-17T19:07:04.0047590Z * [new branch] lan/test_repo_dispatch_circleci -> origin/lan/test_repo_dispatch_circleci 2022-03-17T19:07:04.0048664Z * [new branch] lan/test_repo_dispatch_gha -> origin/lan/test_repo_dispatch_gha 2022-03-17T19:07:04.0049959Z * [new branch] lan/test_reverse_trigger_binary -> origin/lan/test_reverse_trigger_binary 2022-03-17T19:07:04.0051797Z * [new branch] larryliu-scratch -> origin/larryliu-scratch 2022-03-17T19:07:04.0053161Z * [new branch] layer_norm_tbb -> origin/layer_norm_tbb 2022-03-17T19:07:04.0054827Z * [new branch] lazy_tensor_staging -> origin/lazy_tensor_staging 2022-03-17T19:07:04.0056069Z * [new branch] libcudacxx -> origin/libcudacxx 2022-03-17T19:07:04.0057773Z * [new branch] linalg_docs -> origin/linalg_docs 2022-03-17T19:07:04.0059131Z * [new branch] linalg_svd -> origin/linalg_svd 2022-03-17T19:07:04.0060944Z * [new branch] linbinyu-patch-1 -> origin/linbinyu-patch-1 2022-03-17T19:07:04.0062851Z * [new branch] linbinyu-patch-2 -> origin/linbinyu-patch-2 2022-03-17T19:07:04.0064002Z * [new branch] linear_dyn -> origin/linear_dyn 2022-03-17T19:07:04.0065973Z * [new branch] linear_dynamic -> origin/linear_dynamic 2022-03-17T19:07:04.0067158Z * [new branch] linear_qat_lowering -> origin/linear_qat_lowering 2022-03-17T19:07:04.0068804Z * [new branch] lint2 -> origin/lint2 2022-03-17T19:07:04.0070149Z * [new branch] llvmconda -> origin/llvmconda 2022-03-17T19:07:04.0071756Z * [new branch] llvmpip -> origin/llvmpip 2022-03-17T19:07:04.0073098Z * [new branch] load-ucc.so -> origin/load-ucc.so 2022-03-17T19:07:04.0074698Z * [new branch] logcumsumexp -> origin/logcumsumexp 2022-03-17T19:07:04.0075963Z * [new branch] ltc_threshold -> origin/ltc_threshold 2022-03-17T19:07:04.0078229Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2022-03-17T19:07:04.0079595Z * [new branch] lw_c10d_debug_fix -> origin/lw_c10d_debug_fix 2022-03-17T19:07:04.0081189Z * [new branch] macos-conda-llvm -> origin/macos-conda-llvm 2022-03-17T19:07:04.0082738Z * [new branch] main -> origin/main 2022-03-17T19:07:04.0084430Z * [new branch] makslevental-patch-1 -> origin/makslevental-patch-1 2022-03-17T19:07:04.0085835Z * [new branch] makslevental-patch-1-1 -> origin/makslevental-patch-1-1 2022-03-17T19:07:04.0087581Z * [new branch] makslevental-patch-2 -> origin/makslevental-patch-2 2022-03-17T19:07:04.0089190Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2022-03-17T19:07:04.0090564Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2022-03-17T19:07:04.0092265Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2022-03-17T19:07:04.0093985Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2022-03-17T19:07:04.0094987Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2022-03-17T19:07:04.0096936Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2022-03-17T19:07:04.0098949Z * [new branch] malfet/add-lint-as-mandatory-check-for-ghf -> origin/malfet/add-lint-as-mandatory-check-for-ghf 2022-03-17T19:07:04.0099765Z * [new branch] malfet/add-merge-rules -> origin/malfet/add-merge-rules 2022-03-17T19:07:04.0101211Z * [new branch] malfet/clang-tidy-fixes-take-3 -> origin/malfet/clang-tidy-fixes-take-3 2022-03-17T19:07:04.0103011Z * [new branch] malfet/cp-lazy-load -> origin/malfet/cp-lazy-load 2022-03-17T19:07:04.0104221Z * [new branch] malfet/dedup-cpp-aot-test-names -> origin/malfet/dedup-cpp-aot-test-names 2022-03-17T19:07:04.0105486Z * [new branch] malfet/disable-mac-smoke-builds -> origin/malfet/disable-mac-smoke-builds 2022-03-17T19:07:04.0106879Z * [new branch] malfet/fix-asmjit-clang-13-werror -> origin/malfet/fix-asmjit-clang-13-werror 2022-03-17T19:07:04.0108172Z * [new branch] malfet/ghf-add-push-retries -> origin/malfet/ghf-add-push-retries 2022-03-17T19:07:04.0109890Z * [new branch] malfet/leak-python-engine -> origin/malfet/leak-python-engine 2022-03-17T19:07:04.0111613Z * [new branch] malfet/make-dispatch-globals-const -> origin/malfet/make-dispatch-globals-const 2022-03-17T19:07:04.0126081Z * [new branch] malfet/move-cublas-dependency-after-cudnn -> origin/malfet/move-cublas-dependency-after-cudnn 2022-03-17T19:07:04.0127872Z * [new branch] malfet/refactor-GloomakeDeviceFor -> origin/malfet/refactor-GloomakeDeviceFor 2022-03-17T19:07:04.0129151Z * [new branch] malfet/split-cuda-spectral-ops -> origin/malfet/split-cuda-spectral-ops 2022-03-17T19:07:04.0130980Z * [new branch] malfet/test-builder-changes -> origin/malfet/test-builder-changes 2022-03-17T19:07:04.0132175Z * [new branch] malfet/test-builder-m1 -> origin/malfet/test-builder-m1 2022-03-17T19:07:04.0133986Z * [new branch] malfet/try-enabling-AVX2-on-Windows -> origin/malfet/try-enabling-AVX2-on-Windows 2022-03-17T19:07:04.0135138Z * [new branch] malfet/unify-workflow-dispatch -> origin/malfet/unify-workflow-dispatch 2022-03-17T19:07:04.0136969Z * [new branch] malfet/use-std-copy-in-cat-kernel -> origin/malfet/use-std-copy-in-cat-kernel 2022-03-17T19:07:04.0138577Z * [new branch] mark-test-hotfix -> origin/mark-test-hotfix 2022-03-17T19:07:04.0140806Z * [new branch] master -> origin/master 2022-03-17T19:07:04.0142665Z * [new branch] mathtests -> origin/mathtests 2022-03-17T19:07:04.0144279Z * [new branch] matmul_vectors -> origin/matmul_vectors 2022-03-17T19:07:04.0145937Z * [new branch] matrix_exp -> origin/matrix_exp 2022-03-17T19:07:04.0147510Z * [new branch] max_pool_docs -> origin/max_pool_docs 2022-03-17T19:07:04.0148865Z * [new branch] measure_warmup -> origin/measure_warmup 2022-03-17T19:07:04.0151119Z * [new branch] meganset/output_ratio -> origin/meganset/output_ratio 2022-03-17T19:07:04.0152679Z * [new branch] mem_docs -> origin/mem_docs 2022-03-17T19:07:04.0154055Z * [new branch] merge -> origin/merge 2022-03-17T19:07:04.0164380Z * [new branch] merge_cuda_streams -> origin/merge_cuda_streams 2022-03-17T19:07:04.0166241Z * [new branch] merge_eager_jit_cuda_streams -> origin/merge_eager_jit_cuda_streams 2022-03-17T19:07:04.0167806Z * [new branch] merge_script_pdt_api -> origin/merge_script_pdt_api 2022-03-17T19:07:04.0169183Z * [new branch] meta_support_yf225 -> origin/meta_support_yf225 2022-03-17T19:07:04.0170977Z * [new branch] metacompile_ternary_if -> origin/metacompile_ternary_if 2022-03-17T19:07:04.0185157Z * [new branch] mgpu_integration_proto_ops -> origin/mgpu_integration_proto_ops 2022-03-17T19:07:04.0201805Z * [new branch] mgpu_integration_proto_tex -> origin/mgpu_integration_proto_tex 2022-03-17T19:07:04.0219309Z * [new branch] migrate_other_documentation_from_master_to_main -> origin/migrate_other_documentation_from_master_to_main 2022-03-17T19:07:04.0221004Z * [new branch] migrate_torch_master_references_to_main -> origin/migrate_torch_master_references_to_main 2022-03-17T19:07:04.0222853Z * [new branch] mingfeima -> origin/mingfeima 2022-03-17T19:07:04.0224551Z * [new branch] mobile_cpu_perf_benchmark_pr -> origin/mobile_cpu_perf_benchmark_pr 2022-03-17T19:07:04.0226195Z * [new branch] monitored_barrier_fix -> origin/monitored_barrier_fix 2022-03-17T19:07:04.0227530Z * [new branch] monkey_type -> origin/monkey_type 2022-03-17T19:07:04.0229983Z * [new branch] mostafaelhoushi-patch-1 -> origin/mostafaelhoushi-patch-1 2022-03-17T19:07:04.0231598Z * [new branch] mostafaelhoushi-patch-fix-wrap-fn-in-ast-rewriter -> origin/mostafaelhoushi-patch-fix-wrap-fn-in-ast-rewriter 2022-03-17T19:07:04.0232736Z * [new branch] mrshenli-patch-1 -> origin/mrshenli-patch-1 2022-03-17T19:07:04.0234783Z * [new branch] mrshenli-patch-2 -> origin/mrshenli-patch-2 2022-03-17T19:07:04.0236150Z * [new branch] multi_output_0 -> origin/multi_output_0 2022-03-17T19:07:04.0237793Z * [new branch] multinomial_half -> origin/multinomial_half 2022-03-17T19:07:04.0239198Z * [new branch] mvz-nnc-aot-prototype -> origin/mvz-nnc-aot-prototype 2022-03-17T19:07:04.0240907Z * [new branch] mvz-te-cat-fix-1.7 -> origin/mvz-te-cat-fix-1.7 2022-03-17T19:07:04.0242310Z * [new branch] nccl-on-cpu -> origin/nccl-on-cpu 2022-03-17T19:07:04.0243897Z * [new branch] nestedtensor -> origin/nestedtensor 2022-03-17T19:07:04.0245277Z * [new branch] new_add_default_args -> origin/new_add_default_args 2022-03-17T19:07:04.0246951Z * [new branch] new_merge_streams_1 -> origin/new_merge_streams_1 2022-03-17T19:07:04.0248214Z * [new branch] new_metacompile_if -> origin/new_metacompile_if 2022-03-17T19:07:04.0249887Z * [new branch] new_ones -> origin/new_ones 2022-03-17T19:07:04.0251239Z * [new branch] new_script_type -> origin/new_script_type 2022-03-17T19:07:04.0252983Z * [new branch] newest_ut -> origin/newest_ut 2022-03-17T19:07:04.0254884Z * [new branch] ngimel/THCTensor_cleanup -> origin/ngimel/THCTensor_cleanup 2022-03-17T19:07:04.0255997Z * [new branch] ngimel/THC_reduce_cleanup -> origin/ngimel/THC_reduce_cleanup 2022-03-17T19:07:04.0257786Z * [new branch] ngimel/addmm_instr -> origin/ngimel/addmm_instr 2022-03-17T19:07:04.0259003Z * [new branch] ngimel/addr_test -> origin/ngimel/addr_test 2022-03-17T19:07:04.0260566Z * [new branch] ngimel/alias_gen -> origin/ngimel/alias_gen 2022-03-17T19:07:04.0262742Z * [new branch] ngimel/cleanup -> origin/ngimel/cleanup 2022-03-17T19:07:04.0263807Z * [new branch] ngimel/conv_db -> origin/ngimel/conv_db 2022-03-17T19:07:04.0265091Z * [new branch] ngimel/cos_sim -> origin/ngimel/cos_sim 2022-03-17T19:07:04.0266413Z * [new branch] ngimel/cov_errors -> origin/ngimel/cov_errors 2022-03-17T19:07:04.0267813Z * [new branch] ngimel/fix_indexing -> origin/ngimel/fix_indexing 2022-03-17T19:07:04.0269475Z * [new branch] ngimel/fix_unique -> origin/ngimel/fix_unique 2022-03-17T19:07:04.0270567Z * [new branch] ngimel/flip_resubmit -> origin/ngimel/flip_resubmit 2022-03-17T19:07:04.0271934Z * [new branch] ngimel/i0_accscalar -> origin/ngimel/i0_accscalar 2022-03-17T19:07:04.0273268Z * [new branch] ngimel/igamma -> origin/ngimel/igamma 2022-03-17T19:07:04.0274722Z * [new branch] ngimel/launch_check -> origin/ngimel/launch_check 2022-03-17T19:07:04.0275972Z * [new branch] ngimel/layer_norm -> origin/ngimel/layer_norm 2022-03-17T19:07:04.0278080Z * [new branch] ngimel/linalg_docs -> origin/ngimel/linalg_docs 2022-03-17T19:07:04.0291966Z * [new branch] ngimel/loops_reduce -> origin/ngimel/loops_reduce 2022-03-17T19:07:04.0293267Z * [new branch] ngimel/make_tensor_grad -> origin/ngimel/make_tensor_grad 2022-03-17T19:07:04.0295224Z * [new branch] ngimel/map_alloc_rand -> origin/ngimel/map_alloc_rand 2022-03-17T19:07:04.0296365Z * [new branch] ngimel/masked_select -> origin/ngimel/masked_select 2022-03-17T19:07:04.0297763Z * [new branch] ngimel/no_copy_indices -> origin/ngimel/no_copy_indices 2022-03-17T19:07:04.0299183Z * [new branch] ngimel/no_gc_leakcheck -> origin/ngimel/no_gc_leakcheck 2022-03-17T19:07:04.0300741Z * [new branch] ngimel/norm_backward -> origin/ngimel/norm_backward 2022-03-17T19:07:04.0303494Z * [new branch] ngimel/opinfo_aap -> origin/ngimel/opinfo_aap 2022-03-17T19:07:04.0304495Z * [new branch] ngimel/remove_utilscpp -> origin/ngimel/remove_utilscpp 2022-03-17T19:07:04.0305871Z * [new branch] ngimel/renorm_fix -> origin/ngimel/renorm_fix 2022-03-17T19:07:04.0307279Z * [new branch] ngimel/rnn_context -> origin/ngimel/rnn_context 2022-03-17T19:07:04.0308586Z * [new branch] ngimel/s_where -> origin/ngimel/s_where 2022-03-17T19:07:04.0310309Z * [new branch] ngimel/stack_cat -> origin/ngimel/stack_cat 2022-03-17T19:07:04.0311404Z * [new branch] ngimel/std_var -> origin/ngimel/std_var 2022-03-17T19:07:04.0312798Z * [new branch] ngimel/storage_types -> origin/ngimel/storage_types 2022-03-17T19:07:04.0314218Z * [new branch] ngimel/thc_reduce -> origin/ngimel/thc_reduce 2022-03-17T19:07:04.0315482Z * [new branch] ngimel/unify_norm -> origin/ngimel/unify_norm 2022-03-17T19:07:04.0317010Z * [new branch] ngimel/unique_sliced -> origin/ngimel/unique_sliced 2022-03-17T19:07:04.0318346Z * [new branch] ngimel/view_optim -> origin/ngimel/view_optim 2022-03-17T19:07:04.0320056Z * [new branch] ngimel/zeta_acc -> origin/ngimel/zeta_acc 2022-03-17T19:07:04.0321343Z * [new branch] nightly -> origin/nightly 2022-03-17T19:07:04.0323032Z * [new branch] nightlykickoff -> origin/nightlykickoff 2022-03-17T19:07:04.0325135Z * [new branch] nikitaved/bce_loss_2xbackward_fix -> origin/nikitaved/bce_loss_2xbackward_fix 2022-03-17T19:07:04.0326224Z * [new branch] nikitaved/coo_index_select -> origin/nikitaved/coo_index_select 2022-03-17T19:07:04.0327584Z * [new branch] nikitaved/det_backward -> origin/nikitaved/det_backward 2022-03-17T19:07:04.0329023Z * [new branch] nikitaved/det_double_backward_svd_path -> origin/nikitaved/det_double_backward_svd_path 2022-03-17T19:07:04.0330280Z * [new branch] nikitaved/det_ima_cuda -> origin/nikitaved/det_ima_cuda 2022-03-17T19:07:04.0331517Z * [new branch] nikitaved/det_robust_backward -> origin/nikitaved/det_robust_backward 2022-03-17T19:07:04.0332857Z * [new branch] nikitaved/fix_max_unpool_backward -> origin/nikitaved/fix_max_unpool_backward 2022-03-17T19:07:04.0334166Z * [new branch] nikitaved/householder_forward_ad -> origin/nikitaved/householder_forward_ad 2022-03-17T19:07:04.0335622Z * [new branch] nikitaved/householder_product_optimal_backward -> origin/nikitaved/householder_product_optimal_backward 2022-03-17T19:07:04.0337064Z * [new branch] nikitaved/linalg_qr_forward_AD -> origin/nikitaved/linalg_qr_forward_AD 2022-03-17T19:07:04.0339171Z * [new branch] nikitaved/linalg_svd_rank_revealing -> origin/nikitaved/linalg_svd_rank_revealing 2022-03-17T19:07:04.0353039Z * [new branch] nikitaved/lobpcg_backward_ctx -> origin/nikitaved/lobpcg_backward_ctx 2022-03-17T19:07:04.0354622Z * [new branch] nikitaved/lstsq_forward_backward_ad -> origin/nikitaved/lstsq_forward_backward_ad 2022-03-17T19:07:04.0356014Z * [new branch] nikitaved/lu_backward -> origin/nikitaved/lu_backward 2022-03-17T19:07:04.0357686Z * [new branch] nikitaved/lu_comlex_backward -> origin/nikitaved/lu_comlex_backward 2022-03-17T19:07:04.0359057Z * [new branch] nikitaved/lu_forward_ad -> origin/nikitaved/lu_forward_ad 2022-03-17T19:07:04.0360761Z * [new branch] nikitaved/lu_solve_backward -> origin/nikitaved/lu_solve_backward 2022-03-17T19:07:04.0362012Z * [new branch] nikitaved/lu_solve_forward_ad -> origin/nikitaved/lu_solve_forward_ad 2022-03-17T19:07:04.0363741Z * [new branch] nikitaved/lu_unpack_forward_ad -> origin/nikitaved/lu_unpack_forward_ad 2022-03-17T19:07:04.0365129Z * [new branch] nikitaved/make_bin_cross_entropy_diff_wrt_target -> origin/nikitaved/make_bin_cross_entropy_diff_wrt_target 2022-03-17T19:07:04.0366609Z * [new branch] nikitaved/matrix_rank_missing_interfaces -> origin/nikitaved/matrix_rank_missing_interfaces 2022-03-17T19:07:04.0368127Z * [new branch] nikitaved/max_unpool_opinfos_easier_backward -> origin/nikitaved/max_unpool_opinfos_easier_backward 2022-03-17T19:07:04.0369530Z * [new branch] nikitaved/opinfo_max_pool -> origin/nikitaved/opinfo_max_pool 2022-03-17T19:07:04.0371258Z * [new branch] nikitaved/partial_grad_test -> origin/nikitaved/partial_grad_test 2022-03-17T19:07:04.0372496Z * [new branch] nikitaved/pinv_backward -> origin/nikitaved/pinv_backward 2022-03-17T19:07:04.0374342Z * [new branch] nikitaved/remove_index_fill_side_effect_for_scalar_input -> origin/nikitaved/remove_index_fill_side_effect_for_scalar_input 2022-03-17T19:07:04.0375698Z * [new branch] nikitaved/solve_methods_forward_AD -> origin/nikitaved/solve_methods_forward_AD 2022-03-17T19:07:04.0377440Z * [new branch] nikitaved/stable_sort_CPU -> origin/nikitaved/stable_sort_CPU 2022-03-17T19:07:04.0378672Z * [new branch] nikitaved/stable_sort_CPU_take_2 -> origin/nikitaved/stable_sort_CPU_take_2 2022-03-17T19:07:04.0380006Z * [new branch] nikitaved/svd_lowrank_opinfo -> origin/nikitaved/svd_lowrank_opinfo 2022-03-17T19:07:04.0382411Z * [new branch] nikitaved/test_neg_view_det_singular -> origin/nikitaved/test_neg_view_det_singular 2022-03-17T19:07:04.0383582Z * [new branch] nikitha_language_spec -> origin/nikitha_language_spec 2022-03-17T19:07:04.0385507Z * [new branch] nikitha_stringDedantSupport -> origin/nikitha_stringDedantSupport 2022-03-17T19:07:04.0386942Z * [new branch] nikved/cumsum_complex_backward -> origin/nikved/cumsum_complex_backward 2022-03-17T19:07:04.0388659Z * [new branch] nikved/eig_complex_backward -> origin/nikved/eig_complex_backward 2022-03-17T19:07:04.0389890Z * [new branch] nikved/gather_add -> origin/nikved/gather_add 2022-03-17T19:07:04.0391265Z * [new branch] nikved/index_fill_port -> origin/nikved/index_fill_port 2022-03-17T19:07:04.0392652Z * [new branch] nikved/linalg_lstsq -> origin/nikved/linalg_lstsq 2022-03-17T19:07:04.0394402Z * [new branch] nikved/lobpcg_remove_inverse -> origin/nikved/lobpcg_remove_inverse 2022-03-17T19:07:04.0396308Z * [new branch] nikved/native_lu_unpack -> origin/nikved/native_lu_unpack 2022-03-17T19:07:04.0397456Z * [new branch] nikved/stable_sort_CPU -> origin/nikved/stable_sort_CPU 2022-03-17T19:07:04.0399367Z * [new branch] nll_backward_2 -> origin/nll_backward_2 2022-03-17T19:07:04.0401049Z * [new branch] nnapi_log_softmax -> origin/nnapi_log_softmax 2022-03-17T19:07:04.0402215Z * [new branch] no_zerodim_type_promo -> origin/no_zerodim_type_promo 2022-03-17T19:07:04.0404258Z * [new branch] non_overlapping -> origin/non_overlapping 2022-03-17T19:07:04.0405349Z * [new branch] nonzero_opinfo -> origin/nonzero_opinfo 2022-03-17T19:07:04.0407222Z * [new branch] norm -> origin/norm 2022-03-17T19:07:04.0408456Z * [new branch] norm_docs -> origin/norm_docs 2022-03-17T19:07:04.0410284Z * [new branch] normal_fix -> origin/normal_fix 2022-03-17T19:07:04.0411541Z * [new branch] normalization -> origin/normalization 2022-03-17T19:07:04.0413340Z * [new branch] nt_ops -> origin/nt_ops 2022-03-17T19:07:04.0414976Z * [new branch] nuke -> origin/nuke 2022-03-17T19:07:04.0416338Z * [new branch] on_pull_request_conditional -> origin/on_pull_request_conditional 2022-03-17T19:07:04.0418183Z * [new branch] onnx -> origin/onnx 2022-03-17T19:07:04.0419470Z * [new branch] onnx_fix_sum_keepdims -> origin/onnx_fix_sum_keepdims 2022-03-17T19:07:04.0420948Z * [new branch] onnx_local_function_opset_ms -> origin/onnx_local_function_opset_ms 2022-03-17T19:07:04.0423265Z * [new branch] onnx_ms_1 -> origin/onnx_ms_1 2022-03-17T19:07:04.0424379Z * [new branch] onnx_ms_1_1_22 -> origin/onnx_ms_1_1_22 2022-03-17T19:07:04.0426366Z * [new branch] onnx_quant_bias -> origin/onnx_quant_bias 2022-03-17T19:07:04.0427807Z * [new branch] opaque -> origin/opaque 2022-03-17T19:07:04.0429777Z * [new branch] opaque-dispatch -> origin/opaque-dispatch 2022-03-17T19:07:04.0432125Z * [new branch] opinfo/nll_loss -> origin/opinfo/nll_loss 2022-03-17T19:07:04.0432925Z * [new branch] opinfo/one_hot -> origin/opinfo/one_hot 2022-03-17T19:07:04.0434949Z * [new branch] opinfos_margin_loss -> origin/opinfos_margin_loss 2022-03-17T19:07:04.0436050Z * [new branch] opinfos_pooling -> origin/opinfos_pooling 2022-03-17T19:07:04.0437883Z * [new branch] opinfos_unassigned_misc -> origin/opinfos_unassigned_misc 2022-03-17T19:07:04.0440135Z * [new branch] ops_in_python -> origin/ops_in_python 2022-03-17T19:07:04.0441387Z * [new branch] opti -> origin/opti 2022-03-17T19:07:04.0442825Z * [new branch] optimizers_struct_based -> origin/optimizers_struct_based 2022-03-17T19:07:04.0444302Z * [new branch] optimizers_struct_based_v2 -> origin/optimizers_struct_based_v2 2022-03-17T19:07:04.0445780Z * [new branch] optrm -> origin/optrm 2022-03-17T19:07:04.0448673Z * [new branch] orig/lts/release/1.8 -> origin/orig/lts/release/1.8 2022-03-17T19:07:04.0450589Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2022-03-17T19:07:04.0451646Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2022-03-17T19:07:04.0452840Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2022-03-17T19:07:04.0454019Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2022-03-17T19:07:04.0455417Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2022-03-17T19:07:04.0456658Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2022-03-17T19:07:04.0458883Z * [new branch] origin/fmod-remainder-opinfo -> origin/origin/fmod-remainder-opinfo 2022-03-17T19:07:04.0459593Z * [new branch] origin/record-time -> origin/origin/record-time 2022-03-17T19:07:04.0461781Z * [new branch] paddingsame -> origin/paddingsame 2022-03-17T19:07:04.0463628Z * [new branch] patrickkan/test-help-message -> origin/patrickkan/test-help-message 2022-03-17T19:07:04.0465058Z * [new branch] pavithran/flatbuffer_jit_file -> origin/pavithran/flatbuffer_jit_file 2022-03-17T19:07:04.0466492Z * [new branch] pavithran/flatbuffer_mobile_debug_info -> origin/pavithran/flatbuffer_mobile_debug_info 2022-03-17T19:07:04.0471215Z * [new branch] pbelevich-patch-1 -> origin/pbelevich-patch-1 2022-03-17T19:07:04.0471721Z * [new branch] pbelevich-patch-2 -> origin/pbelevich-patch-2 2022-03-17T19:07:04.0472219Z * [new branch] pbelevich-patch-3 -> origin/pbelevich-patch-3 2022-03-17T19:07:04.0472704Z * [new branch] pdt_tutorial -> origin/pdt_tutorial 2022-03-17T19:07:04.0473937Z * [new branch] per_channel_quant -> origin/per_channel_quant 2022-03-17T19:07:04.0475803Z * [new branch] peterjc123-patch-1 -> origin/peterjc123-patch-1 2022-03-17T19:07:04.0477072Z * [new branch] peterjc123-patch-1-1 -> origin/peterjc123-patch-1-1 2022-03-17T19:07:04.0478520Z * [new branch] pg_fix -> origin/pg_fix 2022-03-17T19:07:04.0480519Z * [new branch] pickle-doc-warning -> origin/pickle-doc-warning 2022-03-17T19:07:04.0481724Z * [new branch] poc_remove_sth_from_torch_nn_module -> origin/poc_remove_sth_from_torch_nn_module 2022-03-17T19:07:04.0483113Z * [new branch] portforward_fix -> origin/portforward_fix 2022-03-17T19:07:04.0485305Z * [new branch] postnightly -> origin/postnightly 2022-03-17T19:07:04.0487593Z * [new branch] pr-33000 -> origin/pr-33000 2022-03-17T19:07:04.0489573Z * [new branch] primTorch_aliases -> origin/primTorch_aliases 2022-03-17T19:07:04.0491908Z * [new branch] pritam/dist_autograd_sync -> origin/pritam/dist_autograd_sync 2022-03-17T19:07:04.0493228Z * [new branch] process_commit_update2 -> origin/process_commit_update2 2022-03-17T19:07:04.0495024Z * [new branch] profile -> origin/profile 2022-03-17T19:07:04.0496749Z * [new branch] profiler_docs -> origin/profiler_docs 2022-03-17T19:07:04.0498112Z * [new branch] profiler_fallback -> origin/profiler_fallback 2022-03-17T19:07:04.0500074Z * [new branch] proto_jiterated_complex -> origin/proto_jiterated_complex 2022-03-17T19:07:04.0501641Z * [new branch] protoissue -> origin/protoissue 2022-03-17T19:07:04.0503606Z * [new branch] prototype_proc -> origin/prototype_proc 2022-03-17T19:07:04.0504810Z * [new branch] proxy_tensor -> origin/proxy_tensor 2022-03-17T19:07:04.0506703Z * [new branch] pt_mobile_api_fc_tmp -> origin/pt_mobile_api_fc_tmp 2022-03-17T19:07:04.0508861Z * [new branch] pull/24055/head -> origin/pull/24055/head 2022-03-17T19:07:04.0510254Z * [new branch] python391-annotation_str -> origin/python391-annotation_str 2022-03-17T19:07:04.0511929Z * [new branch] python_mode -> origin/python_mode 2022-03-17T19:07:04.0513253Z * [new branch] python_mode2 -> origin/python_mode2 2022-03-17T19:07:04.0515315Z * [new branch] pytorch-test_xla_ci2 -> origin/pytorch-test_xla_ci2 2022-03-17T19:07:04.0516519Z * [new branch] pytree_fx -> origin/pytree_fx 2022-03-17T19:07:04.0518397Z * [new branch] qnnpack_dynamic -> origin/qnnpack_dynamic 2022-03-17T19:07:04.0519879Z * [new branch] qnnpack_per_channel_quant -> origin/qnnpack_per_channel_quant 2022-03-17T19:07:04.0521966Z * [new branch] quantization_1_3_doc -> origin/quantization_1_3_doc 2022-03-17T19:07:04.0523273Z * [new branch] quantized_hardtanh -> origin/quantized_hardtanh 2022-03-17T19:07:04.0525226Z * [new branch] quick-fix -> origin/quick-fix 2022-03-17T19:07:04.0526405Z * [new branch] random_test -> origin/random_test 2022-03-17T19:07:04.0528331Z * [new branch] random_testing -> origin/random_testing 2022-03-17T19:07:04.0529616Z * [new branch] re-enable-mac-json-uploads -> origin/re-enable-mac-json-uploads 2022-03-17T19:07:04.0531619Z * [new branch] recursive_module_deepcopy -> origin/recursive_module_deepcopy 2022-03-17T19:07:04.0532833Z * [new branch] redo-override-listing -> origin/redo-override-listing 2022-03-17T19:07:04.0534718Z * [new branch] reduction_type_promotion -> origin/reduction_type_promotion 2022-03-17T19:07:04.0536444Z * [new branch] refactor_build_wheel -> origin/refactor_build_wheel 2022-03-17T19:07:04.0537792Z * [new branch] reference_inputs -> origin/reference_inputs 2022-03-17T19:07:04.0540124Z * [new branch] release/1.10 -> origin/release/1.10 2022-03-17T19:07:04.0541485Z * [new branch] release/1.11 -> origin/release/1.11 2022-03-17T19:07:04.0542925Z * [new branch] release/1.11_a -> origin/release/1.11_a 2022-03-17T19:07:04.0544216Z * [new branch] release/1.4 -> origin/release/1.4 2022-03-17T19:07:04.0545464Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2022-03-17T19:07:04.0547741Z * [new branch] release/1.5 -> origin/release/1.5 2022-03-17T19:07:04.0548637Z * [new branch] release/1.6 -> origin/release/1.6 2022-03-17T19:07:04.0550310Z * [new branch] release/1.7 -> origin/release/1.7 2022-03-17T19:07:04.0551216Z * [new branch] release/1.8 -> origin/release/1.8 2022-03-17T19:07:04.0553109Z * [new branch] release/1.9 -> origin/release/1.9 2022-03-17T19:07:04.0554864Z * [new branch] release_1.8_fq_docs_patch -> origin/release_1.8_fq_docs_patch 2022-03-17T19:07:04.0556190Z * [new branch] remove-root-job -> origin/remove-root-job 2022-03-17T19:07:04.0558139Z * [new branch] remove-test-history-script -> origin/remove-test-history-script 2022-03-17T19:07:04.0559294Z * [new branch] remove-win-smoke-tests -> origin/remove-win-smoke-tests 2022-03-17T19:07:04.0561360Z * [new branch] removeAnnoyingWarnings -> origin/removeAnnoyingWarnings 2022-03-17T19:07:04.0562617Z * [new branch] remove_duplicate -> origin/remove_duplicate 2022-03-17T19:07:04.0564517Z * [new branch] remove_legacy_definitions -> origin/remove_legacy_definitions 2022-03-17T19:07:04.0565840Z * [new branch] remove_opinfo_warning -> origin/remove_opinfo_warning 2022-03-17T19:07:04.0567720Z * [new branch] remove_prints -> origin/remove_prints 2022-03-17T19:07:04.0569098Z * [new branch] remove_redundant_print -> origin/remove_redundant_print 2022-03-17T19:07:04.0571033Z * [new branch] remove_test -> origin/remove_test 2022-03-17T19:07:04.0572345Z * [new branch] rename-quant-module -> origin/rename-quant-module 2022-03-17T19:07:04.0574533Z * [new branch] rename-quant-module2 -> origin/rename-quant-module2 2022-03-17T19:07:04.0576439Z * [new branch] rename_index_to_advanced_index -> origin/rename_index_to_advanced_index 2022-03-17T19:07:04.0578584Z * [new branch] resnet_forward_ad_ops -> origin/resnet_forward_ad_ops 2022-03-17T19:07:04.0581021Z * [new branch] revert-55729-gh/msaroufim/5/head -> origin/revert-55729-gh/msaroufim/5/head 2022-03-17T19:07:04.0583333Z * [new branch] revert-62257-deyu/remove_use_external_data_format -> origin/revert-62257-deyu/remove_use_external_data_format 2022-03-17T19:07:04.0585453Z * [new branch] revert-62766-gh/BowenBao/109/head -> origin/revert-62766-gh/BowenBao/109/head 2022-03-17T19:07:04.0588290Z * [new branch] revert-64051-gh/jaceyca/2/head -> origin/revert-64051-gh/jaceyca/2/head 2022-03-17T19:07:04.0590992Z * [new branch] revert-65014-gh/mcr229/1/head -> origin/revert-65014-gh/mcr229/1/head 2022-03-17T19:07:04.0593167Z * [new branch] revert-66735-gh/PaliC/5/head -> origin/revert-66735-gh/PaliC/5/head 2022-03-17T19:07:04.0596044Z * [new branch] revert-69916-gh/wanchaol/193/head -> origin/revert-69916-gh/wanchaol/193/head 2022-03-17T19:07:04.0598458Z * [new branch] revert-69943-revert-69916-gh/wanchaol/193/head -> origin/revert-69943-revert-69916-gh/wanchaol/193/head 2022-03-17T19:07:04.0600869Z * [new branch] revert-73214-gh/H-Huang/47/head -> origin/revert-73214-gh/H-Huang/47/head 2022-03-17T19:07:04.0602193Z * [new branch] revivify_ci -> origin/revivify_ci 2022-03-17T19:07:04.0604186Z * [new branch] rnn_reduce_range -> origin/rnn_reduce_range 2022-03-17T19:07:04.0606462Z * [new branch] robieta/debug_info -> origin/robieta/debug_info 2022-03-17T19:07:04.0607534Z * [new branch] robieta/module_tracking -> origin/robieta/module_tracking 2022-03-17T19:07:04.0608927Z * [new branch] robieta/module_tracking_reland -> origin/robieta/module_tracking_reland 2022-03-17T19:07:04.0610316Z * [new branch] robieta/timer_div_zero -> origin/robieta/timer_div_zero 2022-03-17T19:07:04.0611691Z * [new branch] robieta/torchtidy_prototype -> origin/robieta/torchtidy_prototype 2022-03-17T19:07:04.0613582Z * [new branch] rocm-dist-master -> origin/rocm-dist-master 2022-03-17T19:07:04.0614917Z * [new branch] rocm_gha -> origin/rocm_gha 2022-03-17T19:07:04.0616847Z * [new branch] rocm_slow_test_label -> origin/rocm_slow_test_label 2022-03-17T19:07:04.0618213Z * [new branch] rohan-varma-patch-1 -> origin/rohan-varma-patch-1 2022-03-17T19:07:04.0620150Z * [new branch] rohan-varma-patch-2 -> origin/rohan-varma-patch-2 2022-03-17T19:07:04.0621697Z * [new branch] rohan_fix -> origin/rohan_fix 2022-03-17T19:07:04.0623979Z * [new branch] rpc_dev -> origin/rpc_dev 2022-03-17T19:07:04.0625370Z * [new branch] running-generate_ci_workflows -> origin/running-generate_ci_workflows 2022-03-17T19:07:04.0627204Z * [new branch] saketh-modules-lab -> origin/saketh-modules-lab 2022-03-17T19:07:04.0629636Z * [new branch] samestep/docker-cache-gha -> origin/samestep/docker-cache-gha 2022-03-17T19:07:04.0630754Z * [new branch] samestep/gha-ossci-metrics-new-token -> origin/samestep/gha-ossci-metrics-new-token 2022-03-17T19:07:04.0632154Z * [new branch] samestep/redundant-workflows-base -> origin/samestep/redundant-workflows-base 2022-03-17T19:07:04.0633833Z * [new branch] saved-tensors-docstring -> origin/saved-tensors-docstring 2022-03-17T19:07:04.0635133Z * [new branch] saved_tensors_docs -> origin/saved_tensors_docs 2022-03-17T19:07:04.0636945Z * [new branch] scalarListAPI -> origin/scalarListAPI 2022-03-17T19:07:04.0638313Z * [new branch] scatter-reductions-cpu -> origin/scatter-reductions-cpu 2022-03-17T19:07:04.0640186Z * [new branch] scatter_gather_more_shape_checks -> origin/scatter_gather_more_shape_checks 2022-03-17T19:07:04.0641423Z * [new branch] scatter_gather_to_ATen_CUDA -> origin/scatter_gather_to_ATen_CUDA 2022-03-17T19:07:04.0643677Z * [new branch] sdym/check-default-branch -> origin/sdym/check-default-branch 2022-03-17T19:07:04.0644787Z * [new branch] sdym/fix-54979 -> origin/sdym/fix-54979 2022-03-17T19:07:04.0646082Z * [new branch] sdym/fix-undefined_internal -> origin/sdym/fix-undefined_internal 2022-03-17T19:07:04.0647765Z * [new branch] sdym/generated-stats -> origin/sdym/generated-stats 2022-03-17T19:07:04.0649026Z * [new branch] sdym/gha-android-test -> origin/sdym/gha-android-test 2022-03-17T19:07:04.0650360Z * [new branch] sdym/py310-test_binary_ufuncs.py -> origin/sdym/py310-test_binary_ufuncs.py 2022-03-17T19:07:04.0651772Z * [new branch] sdym/py310-test_distributions -> origin/sdym/py310-test_distributions 2022-03-17T19:07:04.0653089Z * [new branch] sdym/qinterpolate -> origin/sdym/qinterpolate 2022-03-17T19:07:04.0654421Z * [new branch] sdym/test-merge -> origin/sdym/test-merge 2022-03-17T19:07:04.0656249Z * [new branch] sdym/test-mobile-rules -> origin/sdym/test-mobile-rules 2022-03-17T19:07:04.0657658Z * [new branch] seemethere-patch-1 -> origin/seemethere-patch-1 2022-03-17T19:07:04.0659889Z * [new branch] seemethere/actions_wheels -> origin/seemethere/actions_wheels 2022-03-17T19:07:04.0661021Z * [new branch] seemethere/add_core_to_ghf -> origin/seemethere/add_core_to_ghf 2022-03-17T19:07:04.0662647Z * [new branch] seemethere/fix_collect_env -> origin/seemethere/fix_collect_env 2022-03-17T19:07:04.0664044Z * [new branch] seemethere/formalize-patch-release-process -> origin/seemethere/formalize-patch-release-process 2022-03-17T19:07:04.0665288Z * [new branch] seemethere/nightly_pls_work -> origin/seemethere/nightly_pls_work 2022-03-17T19:07:04.0667179Z * [new branch] seemethere/pull_request_target -> origin/seemethere/pull_request_target 2022-03-17T19:07:04.0668324Z * [new branch] seemethere/rebuild_rocm_stable_image -> origin/seemethere/rebuild_rocm_stable_image 2022-03-17T19:07:04.0669654Z * [new branch] seemethere/rebuilding_lts_18_docker_images -> origin/seemethere/rebuilding_lts_18_docker_images 2022-03-17T19:07:04.0671631Z * [new branch] seemethere/refactor_build_wheel -> origin/seemethere/refactor_build_wheel 2022-03-17T19:07:04.0672775Z * [new branch] seemethere/rocm_not_on_master -> origin/seemethere/rocm_not_on_master 2022-03-17T19:07:04.0674840Z * [new branch] seemethere/test_collect_env -> origin/seemethere/test_collect_env 2022-03-17T19:07:04.0702664Z * [new branch] seemethere/update_branch_cut -> origin/seemethere/update_branch_cut 2022-03-17T19:07:04.0704044Z * [new branch] serialize_undefined_tensor -> origin/serialize_undefined_tensor 2022-03-17T19:07:04.0706118Z * [new branch] set_default_dtype_docs -> origin/set_default_dtype_docs 2022-03-17T19:07:04.0708394Z * [new branch] shape_info -> origin/shape_info 2022-03-17T19:07:04.0709721Z * [new branch] shen-fixes -> origin/shen-fixes 2022-03-17T19:07:04.0711847Z * [new branch] side_stream_patch -> origin/side_stream_patch 2022-03-17T19:07:04.0713158Z * [new branch] silent-rm-error -> origin/silent-rm-error 2022-03-17T19:07:04.0715176Z * [new branch] size_investigation -> origin/size_investigation 2022-03-17T19:07:04.0716546Z * [new branch] skip_meta_conversion -> origin/skip_meta_conversion 2022-03-17T19:07:04.0718469Z * [new branch] sln -> origin/sln 2022-03-17T19:07:04.0720425Z * [new branch] smessmer/lto -> origin/smessmer/lto 2022-03-17T19:07:04.0721493Z * [new branch] smessmer/scalar -> origin/smessmer/scalar 2022-03-17T19:07:04.0723401Z * [new branch] sort-cxx11 -> origin/sort-cxx11 2022-03-17T19:07:04.0724743Z * [new branch] sort-segment -> origin/sort-segment 2022-03-17T19:07:04.0727242Z * [new branch] soulitzer-patch-1 -> origin/soulitzer-patch-1 2022-03-17T19:07:04.0728554Z * [new branch] soulitzer/benchmarking -> origin/soulitzer/benchmarking 2022-03-17T19:07:04.0730244Z * [new branch] source_range -> origin/source_range 2022-03-17T19:07:04.0731605Z * [new branch] sparse-csr-layout -> origin/sparse-csr-layout 2022-03-17T19:07:04.0733498Z * [new branch] sparse-error-msgs -> origin/sparse-error-msgs 2022-03-17T19:07:04.0734682Z * [new branch] sparse_rpc_fix -> origin/sparse_rpc_fix 2022-03-17T19:07:04.0736565Z * [new branch] stack2 -> origin/stack2 2022-03-17T19:07:04.0737882Z * [new branch] stale -> origin/stale 2022-03-17T19:07:04.0739782Z * [new branch] static_dispatch -> origin/static_dispatch 2022-03-17T19:07:04.0741001Z * [new branch] stream_patch -> origin/stream_patch 2022-03-17T19:07:04.0743068Z * [new branch] streamcontext_nop -> origin/streamcontext_nop 2022-03-17T19:07:04.0744304Z * [new branch] stringDedentSupport -> origin/stringDedentSupport 2022-03-17T19:07:04.0746121Z * [new branch] string_view -> origin/string_view 2022-03-17T19:07:04.0747455Z * [new branch] structured_amax -> origin/structured_amax 2022-03-17T19:07:04.0749469Z * [new branch] structured_mse_loss -> origin/structured_mse_loss 2022-03-17T19:07:04.0750730Z * [new branch] su_ghf -> origin/su_ghf 2022-03-17T19:07:04.0752853Z * [new branch] sub_iter_strides_cuda -> origin/sub_iter_strides_cuda 2022-03-17T19:07:04.0754140Z * [new branch] suo-patch-1 -> origin/suo-patch-1 2022-03-17T19:07:04.0756056Z * [new branch] suo-patch-2 -> origin/suo-patch-2 2022-03-17T19:07:04.0757329Z * [new branch] suo-patch-8 -> origin/suo-patch-8 2022-03-17T19:07:04.0759311Z * [new branch] sup -> origin/sup 2022-03-17T19:07:04.0760674Z * [new branch] support_any -> origin/support_any 2022-03-17T19:07:04.0762606Z * [new branch] support_complex -> origin/support_complex 2022-03-17T19:07:04.0763829Z * [new branch] support_pow -> origin/support_pow 2022-03-17T19:07:04.0765692Z * [new branch] support_sum -> origin/support_sum 2022-03-17T19:07:04.0767184Z * [new branch] support_type_inference_nn_modules -> origin/support_type_inference_nn_modules 2022-03-17T19:07:04.0768847Z * [new branch] suraj813-patch-1 -> origin/suraj813-patch-1 2022-03-17T19:07:04.0770479Z * [new branch] svdvals -> origin/svdvals 2022-03-17T19:07:04.0772302Z * [new branch] syncbatchnorm-0-batch-dim -> origin/syncbatchnorm-0-batch-dim 2022-03-17T19:07:04.0773506Z * [new branch] t-opinfo -> origin/t-opinfo 2022-03-17T19:07:04.0775458Z * [new branch] templated-boxed-fallback -> origin/templated-boxed-fallback 2022-03-17T19:07:04.0776746Z * [new branch] tensor-iterator-manual -> origin/tensor-iterator-manual 2022-03-17T19:07:04.0778642Z * [new branch] tensor_docs -> origin/tensor_docs 2022-03-17T19:07:04.0780429Z * [new branch] tensor_proto -> origin/tensor_proto 2022-03-17T19:07:04.0782077Z * [new branch] tensordot_bug -> origin/tensordot_bug 2022-03-17T19:07:04.0784300Z * [new branch] test-master -> origin/test-master 2022-03-17T19:07:04.0785641Z * [new branch] test-parse-ref-on-win -> origin/test-parse-ref-on-win 2022-03-17T19:07:04.0787786Z * [new branch] test_35736 -> origin/test_35736 2022-03-17T19:07:04.0788867Z * [new branch] test_deploy -> origin/test_deploy 2022-03-17T19:07:04.0790894Z * [new branch] test_equiv_recurrent_disable -> origin/test_equiv_recurrent_disable 2022-03-17T19:07:04.0792190Z * [new branch] test_flaky -> origin/test_flaky 2022-03-17T19:07:04.0794478Z * [new branch] test_functional_non_const_ref -> origin/test_functional_non_const_ref 2022-03-17T19:07:04.0795429Z * [new branch] test_k_0312 -> origin/test_k_0312 2022-03-17T19:07:04.0796872Z * [new branch] test_no_warns -> origin/test_no_warns 2022-03-17T19:07:04.0798840Z * [new branch] test_ops_to_support_iterables -> origin/test_ops_to_support_iterables 2022-03-17T19:07:04.0815242Z * [new branch] test_pr -> origin/test_pr 2022-03-17T19:07:04.0816167Z * [new branch] test_rocm_35959 -> origin/test_rocm_35959 2022-03-17T19:07:04.0816978Z * [new branch] test_xla_codegen_changes -> origin/test_xla_codegen_changes 2022-03-17T19:07:04.0817364Z * [new branch] testing_lint_fix -> origin/testing_lint_fix 2022-03-17T19:07:04.0817799Z * [new branch] tf32 -> origin/tf32 2022-03-17T19:07:04.0818349Z * [new branch] thread-local-default-hooks -> origin/thread-local-default-hooks 2022-03-17T19:07:04.0819357Z * [new branch] tktrungna-patch-1 -> origin/tktrungna-patch-1 2022-03-17T19:07:04.0819764Z * [new branch] tmp-codeowner -> origin/tmp-codeowner 2022-03-17T19:07:04.0820750Z * [new branch] torch_version_update -> origin/torch_version_update 2022-03-17T19:07:04.0821239Z * [new branch] trace_fn_calls2 -> origin/trace_fn_calls2 2022-03-17T19:07:04.0821585Z * [new branch] triangular_solve -> origin/triangular_solve 2022-03-17T19:07:04.0822526Z * [new branch] triangular_solve_permute -> origin/triangular_solve_permute 2022-03-17T19:07:04.0823109Z * [new branch] try-hacking-unique -> origin/try-hacking-unique 2022-03-17T19:07:04.0823652Z * [new branch] try-retryable-steps -> origin/try-retryable-steps 2022-03-17T19:07:04.0824459Z * [new branch] tt_emb -> origin/tt_emb 2022-03-17T19:07:04.0825925Z * [new branch] turn_on_cpp_api_parity_test -> origin/turn_on_cpp_api_parity_test 2022-03-17T19:07:04.0828186Z * [new branch] turn_on_cpp_api_parity_test_new_new -> origin/turn_on_cpp_api_parity_test_new_new 2022-03-17T19:07:04.0829463Z * [new branch] turn_on_cpp_api_parity_test_v2 -> origin/turn_on_cpp_api_parity_test_v2 2022-03-17T19:07:04.0831086Z * [new branch] turn_on_cpp_api_parity_test_v3 -> origin/turn_on_cpp_api_parity_test_v3 2022-03-17T19:07:04.0904877Z * [new branch] turn_on_cpp_api_parity_test_v3_exp1 -> origin/turn_on_cpp_api_parity_test_v3_exp1 2022-03-17T19:07:04.1168881Z * [new branch] turn_on_cpp_api_parity_test_v3_exp2 -> origin/turn_on_cpp_api_parity_test_v3_exp2 2022-03-17T19:07:04.1170359Z * [new branch] type_system_1 -> origin/type_system_1 2022-03-17T19:07:04.1171986Z * [new branch] ucc -> origin/ucc 2022-03-17T19:07:04.1173840Z * [new branch] ucc-build -> origin/ucc-build 2022-03-17T19:07:04.1175193Z * [new branch] ucc-test- -> origin/ucc-test- 2022-03-17T19:07:04.1176738Z * [new branch] unbreak_bc -> origin/unbreak_bc 2022-03-17T19:07:04.1178605Z * [new branch] unique_opinfo -> origin/unique_opinfo 2022-03-17T19:07:04.1179863Z * [new branch] upd_k_0609 -> origin/upd_k_0609 2022-03-17T19:07:04.1181817Z * [new branch] upd_kineto_0324 -> origin/upd_kineto_0324 2022-03-17T19:07:04.1183292Z * [new branch] upd_kineto_0330 -> origin/upd_kineto_0330 2022-03-17T19:07:04.1184849Z * [new branch] upd_kineto_0804 -> origin/upd_kineto_0804 2022-03-17T19:07:04.1186415Z * [new branch] upd_kineto_rev_4 -> origin/upd_kineto_rev_4 2022-03-17T19:07:04.1187992Z * [new branch] update-build-bin -> origin/update-build-bin 2022-03-17T19:07:04.1190029Z * [new branch] update-stale-1 -> origin/update-stale-1 2022-03-17T19:07:04.1191401Z * [new branch] update-test-distribute -> origin/update-test-distribute 2022-03-17T19:07:04.1193014Z * [new branch] update-test-libtorch -> origin/update-test-libtorch 2022-03-17T19:07:04.1194607Z * [new branch] update-test-libtorch-path -> origin/update-test-libtorch-path 2022-03-17T19:07:04.1196592Z * [new branch] update-test-path -> origin/update-test-path 2022-03-17T19:07:04.1197953Z * [new branch] update-test-vulkan-path -> origin/update-test-vulkan-path 2022-03-17T19:07:04.1199352Z * [new branch] update_gloo_04152021 -> origin/update_gloo_04152021 2022-03-17T19:07:04.1200874Z * [new branch] update_gloo_submodule -> origin/update_gloo_submodule 2022-03-17T19:07:04.1202422Z * [new branch] update_kineto_0505 -> origin/update_kineto_0505 2022-03-17T19:07:04.1203981Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2022-03-17T19:07:04.1205511Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2022-03-17T19:07:04.1207046Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2022-03-17T19:07:04.1209438Z * [new branch] upstream/ci-all/record-time-slowtests-krshrimali -> origin/upstream/ci-all/record-time-slowtests-krshrimali 2022-03-17T19:07:04.1210633Z * [new branch] upstream/nikved/linalg_lstsq -> origin/upstream/nikved/linalg_lstsq 2022-03-17T19:07:04.1212224Z * [new branch] v0.1.2 -> origin/v0.1.2 2022-03-17T19:07:04.1214137Z * [new branch] v1.0.1 -> origin/v1.0.1 2022-03-17T19:07:04.1215470Z * [new branch] v1.0.3 -> origin/v1.0.3 2022-03-17T19:07:04.1217118Z * [new branch] v1.1.0 -> origin/v1.1.0 2022-03-17T19:07:04.1219174Z * [new branch] v1.2.0 -> origin/v1.2.0 2022-03-17T19:07:04.1220649Z * [new branch] v1.3.0 -> origin/v1.3.0 2022-03-17T19:07:04.1222500Z * [new branch] v1.3.1 -> origin/v1.3.1 2022-03-17T19:07:04.1224700Z * [new branch] v1.4 -> origin/v1.4 2022-03-17T19:07:04.1226095Z * [new branch] version_mismatch_error_msg -> origin/version_mismatch_error_msg 2022-03-17T19:07:04.1228206Z * [new branch] vfdev-5/docs-typo-fix -> origin/vfdev-5/docs-typo-fix 2022-03-17T19:07:04.1229650Z * [new branch] viab -> origin/viab 2022-03-17T19:07:04.1231805Z * [new branch] viable/strict -> origin/viable/strict 2022-03-17T19:07:04.1233033Z * [new branch] view_as_real -> origin/view_as_real 2022-03-17T19:07:04.1234933Z * [new branch] vision_maskrcnn -> origin/vision_maskrcnn 2022-03-17T19:07:04.1236281Z * [new branch] voznesenskym/named_buffers -> origin/voznesenskym/named_buffers 2022-03-17T19:07:04.1237633Z * [new branch] voznesenskym/named_parameters -> origin/voznesenskym/named_parameters 2022-03-17T19:07:04.1239608Z * [new branch] vs-show-message-on-exception -> origin/vs-show-message-on-exception 2022-03-17T19:07:04.1241049Z * [new branch] vsb_cache_aware_benchmark_preview -> origin/vsb_cache_aware_benchmark_preview 2022-03-17T19:07:04.1242971Z * [new branch] vulkan_add_hardshrink_op -> origin/vulkan_add_hardshrink_op 2022-03-17T19:07:04.1244217Z * [new branch] vulkan_conv2d_old_prepack -> origin/vulkan_conv2d_old_prepack 2022-03-17T19:07:04.1245828Z * [new branch] vulkan_conv_cpushader -> origin/vulkan_conv_cpushader 2022-03-17T19:07:04.1247288Z * [new branch] vulkan_frcnn_ops -> origin/vulkan_frcnn_ops 2022-03-17T19:07:04.1249220Z * [new branch] walterddr-bc-checker-hotfix -> origin/walterddr-bc-checker-hotfix 2022-03-17T19:07:04.1250851Z * [new branch] wconstab/backend -> origin/wconstab/backend 2022-03-17T19:07:04.1252102Z * [new branch] wconstab/bench -> origin/wconstab/bench 2022-03-17T19:07:04.1253545Z * [new branch] wconstab/bench2 -> origin/wconstab/bench2 2022-03-17T19:07:04.1254820Z * [new branch] wconstab/benchit -> origin/wconstab/benchit 2022-03-17T19:07:04.1256122Z * [new branch] wconstab/benchmark -> origin/wconstab/benchmark 2022-03-17T19:07:04.1257529Z * [new branch] wconstab/const_containerhash -> origin/wconstab/const_containerhash 2022-03-17T19:07:04.1258795Z * [new branch] wconstab/dbg_nll -> origin/wconstab/dbg_nll 2022-03-17T19:07:04.1260113Z * [new branch] wconstab/dcgan_ops -> origin/wconstab/dcgan_ops 2022-03-17T19:07:04.1262103Z * [new branch] wconstab/debug -> origin/wconstab/debug 2022-03-17T19:07:04.1263028Z * [new branch] wconstab/devdata -> origin/wconstab/devdata 2022-03-17T19:07:04.1264609Z * [new branch] wconstab/example.py_ci -> origin/wconstab/example.py_ci 2022-03-17T19:07:04.1266000Z * [new branch] wconstab/fallback -> origin/wconstab/fallback 2022-03-17T19:07:04.1267903Z * [new branch] wconstab/fallback2 -> origin/wconstab/fallback2 2022-03-17T19:07:04.1268856Z * [new branch] wconstab/fields -> origin/wconstab/fields 2022-03-17T19:07:04.1270302Z * [new branch] wconstab/fix_clone -> origin/wconstab/fix_clone 2022-03-17T19:07:04.1271597Z * [new branch] wconstab/hash_perf -> origin/wconstab/hash_perf 2022-03-17T19:07:04.1273730Z * [new branch] wconstab/improve_deploy_cpy_build -> origin/wconstab/improve_deploy_cpy_build 2022-03-17T19:07:04.1292141Z * [new branch] wconstab/incorrect -> origin/wconstab/incorrect 2022-03-17T19:07:04.1304536Z * [new branch] wconstab/local_scalar_dense -> origin/wconstab/local_scalar_dense 2022-03-17T19:07:04.1306650Z * [new branch] wconstab/lr_hack -> origin/wconstab/lr_hack 2022-03-17T19:07:04.1308702Z * [new branch] wconstab/ltc-noopt -> origin/wconstab/ltc-noopt 2022-03-17T19:07:04.1310704Z * [new branch] wconstab/ltc-nvfuser-nofallback -> origin/wconstab/ltc-nvfuser-nofallback 2022-03-17T19:07:04.1312134Z * [new branch] wconstab/ltc_codegen -> origin/wconstab/ltc_codegen 2022-03-17T19:07:04.1314654Z * [new branch] wconstab/ltc_codegen_clone -> origin/wconstab/ltc_codegen_clone 2022-03-17T19:07:04.1316588Z * [new branch] wconstab/ltc_core_hash -> origin/wconstab/ltc_core_hash 2022-03-17T19:07:04.1318628Z * [new branch] wconstab/move_ltc_in_tree -> origin/wconstab/move_ltc_in_tree 2022-03-17T19:07:04.1320448Z * [new branch] wconstab/mul -> origin/wconstab/mul 2022-03-17T19:07:04.1322956Z * [new branch] wconstab/native_batch_norm -> origin/wconstab/native_batch_norm 2022-03-17T19:07:04.1324874Z * [new branch] wconstab/node_base -> origin/wconstab/node_base 2022-03-17T19:07:04.1326727Z * [new branch] wconstab/perf -> origin/wconstab/perf 2022-03-17T19:07:04.1328811Z * [new branch] wconstab/refactor_benchmark -> origin/wconstab/refactor_benchmark 2022-03-17T19:07:04.1330282Z * [new branch] wconstab/registrar -> origin/wconstab/registrar 2022-03-17T19:07:04.1332445Z * [new branch] wconstab/remove_hash -> origin/wconstab/remove_hash 2022-03-17T19:07:04.1334339Z * [new branch] wconstab/run_tb -> origin/wconstab/run_tb 2022-03-17T19:07:04.1336180Z * [new branch] wconstab/scalar -> origin/wconstab/scalar 2022-03-17T19:07:04.1338042Z * [new branch] wconstab/shape_cache -> origin/wconstab/shape_cache 2022-03-17T19:07:04.1339899Z * [new branch] wconstab/shape_hash -> origin/wconstab/shape_hash 2022-03-17T19:07:04.1342080Z * [new branch] wconstab/shapeinfr -> origin/wconstab/shapeinfr 2022-03-17T19:07:04.1344730Z * [new branch] wconstab/structured_native_batch_norm -> origin/wconstab/structured_native_batch_norm 2022-03-17T19:07:04.1346374Z * [new branch] wconstab/torch_python_objs -> origin/wconstab/torch_python_objs 2022-03-17T19:07:04.1348665Z * [new branch] wconstab/torchpy -> origin/wconstab/torchpy 2022-03-17T19:07:04.1350787Z * [new branch] win_clang -> origin/win_clang 2022-03-17T19:07:04.1353023Z * [new branch] windows_distributions_tests -> origin/windows_distributions_tests 2022-03-17T19:07:04.1355125Z * [new branch] workflow-per-job -> origin/workflow-per-job 2022-03-17T19:07:04.1357482Z * [new branch] xz9/add-release-testing -> origin/xz9/add-release-testing 2022-03-17T19:07:04.1358998Z * [new branch] xz9/fix-tb-branch-run -> origin/xz9/fix-tb-branch-run 2022-03-17T19:07:04.1361019Z * [new branch] xz9/fix-tb-ci -> origin/xz9/fix-tb-ci 2022-03-17T19:07:04.1362896Z * [new branch] xz9/fix-torchbench-ci-update -> origin/xz9/fix-torchbench-ci-update 2022-03-17T19:07:04.1364947Z * [new branch] xz9/test-ghi-68086 -> origin/xz9/test-ghi-68086 2022-03-17T19:07:04.1366443Z * [new branch] xz9/test-pr-71710 -> origin/xz9/test-pr-71710 2022-03-17T19:07:04.1368559Z * [new branch] xz9/torchbench-test -> origin/xz9/torchbench-test 2022-03-17T19:07:04.1370188Z * [new branch] xz9/verify_ngimel_layer_norm -> origin/xz9/verify_ngimel_layer_norm 2022-03-17T19:07:04.1372127Z * [new branch] xz9/verify_ngimel_layer_norm_2 -> origin/xz9/verify_ngimel_layer_norm_2 2022-03-17T19:07:04.1374487Z * [new branch] xzhao9/update-docker-build -> origin/xzhao9/update-docker-build 2022-03-17T19:07:04.1376641Z * [new branch] yinsun-41101 -> origin/yinsun-41101 2022-03-17T19:07:04.1379017Z * [new branch] zasdfgbnm-patch-4 -> origin/zasdfgbnm-patch-4 2022-03-17T19:07:04.1381367Z * [new branch] zasdfgbnm/gelu -> origin/zasdfgbnm/gelu 2022-03-17T19:07:04.1383732Z * [new branch] zhouzhuojie-patch-1 -> origin/zhouzhuojie-patch-1 2022-03-17T19:07:04.1386090Z * [new branch] zz/add-github.com-noproxy -> origin/zz/add-github.com-noproxy 2022-03-17T19:07:04.1387693Z * [new tag] ciflow/all/63212 -> ciflow/all/63212 2022-03-17T19:07:04.1389004Z * [new tag] ciflow/all/64387 -> ciflow/all/64387 2022-03-17T19:07:04.1390363Z * [new tag] ciflow/all/67022 -> ciflow/all/67022 2022-03-17T19:07:04.1391715Z * [new tag] ciflow/all/67023 -> ciflow/all/67023 2022-03-17T19:07:04.1393122Z * [new tag] ciflow/all/67079 -> ciflow/all/67079 2022-03-17T19:07:04.1394455Z * [new tag] ciflow/all/67343 -> ciflow/all/67343 2022-03-17T19:07:04.1395831Z * [new tag] ciflow/all/67833 -> ciflow/all/67833 2022-03-17T19:07:04.1397351Z * [new tag] ciflow/all/68328 -> ciflow/all/68328 2022-03-17T19:07:04.1398561Z * [new tag] ciflow/all/69211 -> ciflow/all/69211 2022-03-17T19:07:04.1400255Z * [new tag] ciflow/all/69754 -> ciflow/all/69754 2022-03-17T19:07:04.1401690Z * [new tag] ciflow/all/69828 -> ciflow/all/69828 2022-03-17T19:07:04.1402973Z * [new tag] ciflow/all/70610 -> ciflow/all/70610 2022-03-17T19:07:04.1404697Z * [new tag] ciflow/all/70978 -> ciflow/all/70978 2022-03-17T19:07:04.1406165Z * [new tag] ciflow/all/70979 -> ciflow/all/70979 2022-03-17T19:07:04.1407674Z * [new tag] ciflow/all/70980 -> ciflow/all/70980 2022-03-17T19:07:04.1409474Z * [new tag] ciflow/all/70981 -> ciflow/all/70981 2022-03-17T19:07:04.1410856Z * [new tag] ciflow/all/70982 -> ciflow/all/70982 2022-03-17T19:07:04.1412893Z * [new tag] ciflow/all/70986 -> ciflow/all/70986 2022-03-17T19:07:04.1414155Z * [new tag] ciflow/all/70988 -> ciflow/all/70988 2022-03-17T19:07:04.1415637Z * [new tag] ciflow/all/70989 -> ciflow/all/70989 2022-03-17T19:07:04.1417503Z * [new tag] ciflow/all/71399 -> ciflow/all/71399 2022-03-17T19:07:04.1418995Z * [new tag] ciflow/all/71582 -> ciflow/all/71582 2022-03-17T19:07:04.1420825Z * [new tag] ciflow/all/71670 -> ciflow/all/71670 2022-03-17T19:07:04.1422207Z * [new tag] ciflow/all/72364 -> ciflow/all/72364 2022-03-17T19:07:04.1424309Z * [new tag] ciflow/all/72593 -> ciflow/all/72593 2022-03-17T19:07:04.1443137Z * [new tag] ciflow/all/72710 -> ciflow/all/72710 2022-03-17T19:07:04.1444228Z * [new tag] ciflow/all/73202 -> ciflow/all/73202 2022-03-17T19:07:04.1446069Z * [new tag] ciflow/all/73403 -> ciflow/all/73403 2022-03-17T19:07:04.1447140Z * [new tag] ciflow/all/73627 -> ciflow/all/73627 2022-03-17T19:07:04.1448323Z * [new tag] ciflow/all/73660 -> ciflow/all/73660 2022-03-17T19:07:04.1449611Z * [new tag] ciflow/all/74111 -> ciflow/all/74111 2022-03-17T19:07:04.1450552Z * [new tag] ciflow/all/74226 -> ciflow/all/74226 2022-03-17T19:07:04.1451777Z * [new tag] ciflow/binaries/71893 -> ciflow/binaries/71893 2022-03-17T19:07:04.1452834Z * [new tag] ciflow/binaries/73342 -> ciflow/binaries/73342 2022-03-17T19:07:04.1453668Z * [new tag] ciflow/binaries/73393 -> ciflow/binaries/73393 2022-03-17T19:07:04.1454752Z * [new tag] ciflow/binaries/74141 -> ciflow/binaries/74141 2022-03-17T19:07:04.1456022Z * [new tag] ciflow/cpu/67093 -> ciflow/cpu/67093 2022-03-17T19:07:04.1456730Z * [new tag] ciflow/cpu/67094 -> ciflow/cpu/67094 2022-03-17T19:07:04.1457622Z * [new tag] ciflow/cpu/67096 -> ciflow/cpu/67096 2022-03-17T19:07:04.1458606Z * [new tag] ciflow/cpu/71399 -> ciflow/cpu/71399 2022-03-17T19:07:04.1459686Z * [new tag] ciflow/cpu/73342 -> ciflow/cpu/73342 2022-03-17T19:07:04.1461491Z * [new tag] ciflow/cpu/74294 -> ciflow/cpu/74294 2022-03-17T19:07:04.1462357Z * [new tag] ciflow/cuda/69828 -> ciflow/cuda/69828 2022-03-17T19:07:04.1463104Z * [new tag] ciflow/cuda/70978 -> ciflow/cuda/70978 2022-03-17T19:07:04.1463992Z * [new tag] ciflow/cuda/70979 -> ciflow/cuda/70979 2022-03-17T19:07:04.1465101Z * [new tag] ciflow/cuda/70980 -> ciflow/cuda/70980 2022-03-17T19:07:04.1465944Z * [new tag] ciflow/cuda/70981 -> ciflow/cuda/70981 2022-03-17T19:07:04.1466967Z * [new tag] ciflow/cuda/70982 -> ciflow/cuda/70982 2022-03-17T19:07:04.1467637Z * [new tag] ciflow/cuda/70986 -> ciflow/cuda/70986 2022-03-17T19:07:04.1468559Z * [new tag] ciflow/cuda/70988 -> ciflow/cuda/70988 2022-03-17T19:07:04.1469553Z * [new tag] ciflow/cuda/70989 -> ciflow/cuda/70989 2022-03-17T19:07:04.1470660Z * [new tag] ciflow/cuda/71213 -> ciflow/cuda/71213 2022-03-17T19:07:04.1471413Z * [new tag] ciflow/cuda/71214 -> ciflow/cuda/71214 2022-03-17T19:07:04.1472257Z * [new tag] ciflow/cuda/71215 -> ciflow/cuda/71215 2022-03-17T19:07:04.1473234Z * [new tag] ciflow/cuda/71216 -> ciflow/cuda/71216 2022-03-17T19:07:04.1474243Z * [new tag] ciflow/cuda/71399 -> ciflow/cuda/71399 2022-03-17T19:07:04.1475534Z * [new tag] ciflow/cuda/74294 -> ciflow/cuda/74294 2022-03-17T19:07:04.1476330Z * [new tag] ciflow/cuda/74343 -> ciflow/cuda/74343 2022-03-17T19:07:04.1477690Z * [new tag] ciflow/ios/72666 -> ciflow/ios/72666 2022-03-17T19:07:04.1479047Z * [new tag] ciflow/libtorch/73011 -> ciflow/libtorch/73011 2022-03-17T19:07:04.1480040Z * [new tag] ciflow/macos/73849 -> ciflow/macos/73849 2022-03-17T19:07:04.1481137Z * [new tag] ciflow/macos/73857 -> ciflow/macos/73857 2022-03-17T19:07:04.1482278Z * [new tag] ciflow/none/73275 -> ciflow/none/73275 2022-03-17T19:07:04.1483350Z * [new tag] ciflow/periodic/74306 -> ciflow/periodic/74306 2022-03-17T19:07:04.1484638Z * [new tag] ciflow/slow-gradcheck/71126 -> ciflow/slow-gradcheck/71126 2022-03-17T19:07:04.1485422Z * [new tag] ciflow/slow-gradcheck/73748 -> ciflow/slow-gradcheck/73748 2022-03-17T19:07:04.1486393Z * [new tag] ciflow/slow/71126 -> ciflow/slow/71126 2022-03-17T19:07:04.1487209Z * [new tag] ciflow/slow/72686 -> ciflow/slow/72686 2022-03-17T19:07:04.1488167Z * [new tag] ciflow/slow/73627 -> ciflow/slow/73627 2022-03-17T19:07:04.1489192Z * [new tag] ciflow/slow/73748 -> ciflow/slow/73748 2022-03-17T19:07:04.1490604Z * [new tag] ciflow/trunk/73788 -> ciflow/trunk/73788 2022-03-17T19:07:04.1491379Z * [new tag] ciflow/trunk/74306 -> ciflow/trunk/74306 2022-03-17T19:07:04.1492876Z * [new tag] nightly-binary -> nightly-binary 2022-03-17T19:07:04.1493977Z * [new tag] v0.1.1 -> v0.1.1 2022-03-17T19:07:04.1495127Z * [new tag] v0.1.10 -> v0.1.10 2022-03-17T19:07:04.1496265Z * [new tag] v0.1.11 -> v0.1.11 2022-03-17T19:07:04.1497358Z * [new tag] v0.1.12 -> v0.1.12 2022-03-17T19:07:04.1498439Z * [new tag] v0.1.2 -> v0.1.2 2022-03-17T19:07:04.1499533Z * [new tag] v0.1.3 -> v0.1.3 2022-03-17T19:07:04.1500799Z * [new tag] v0.1.4 -> v0.1.4 2022-03-17T19:07:04.1502017Z * [new tag] v0.1.5 -> v0.1.5 2022-03-17T19:07:04.1503247Z * [new tag] v0.1.6 -> v0.1.6 2022-03-17T19:07:04.1504303Z * [new tag] v0.1.7 -> v0.1.7 2022-03-17T19:07:04.1505352Z * [new tag] v0.1.8 -> v0.1.8 2022-03-17T19:07:04.1506511Z * [new tag] v0.1.9 -> v0.1.9 2022-03-17T19:07:04.1507643Z * [new tag] v0.2.0 -> v0.2.0 2022-03-17T19:07:04.1508967Z * [new tag] v0.3.0 -> v0.3.0 2022-03-17T19:07:04.1510091Z * [new tag] v0.3.1 -> v0.3.1 2022-03-17T19:07:04.1511230Z * [new tag] v0.4.0 -> v0.4.0 2022-03-17T19:07:04.1512328Z * [new tag] v0.4.1 -> v0.4.1 2022-03-17T19:07:04.1513466Z * [new tag] v1.0.0 -> v1.0.0 2022-03-17T19:07:04.1514670Z * [new tag] v1.0.0a0 -> v1.0.0a0 2022-03-17T19:07:04.1516160Z * [new tag] v1.0.1 -> v1.0.1 2022-03-17T19:07:04.1529998Z * [new tag] v1.0rc0 -> v1.0rc0 2022-03-17T19:07:04.1530654Z * [new tag] v1.0rc1 -> v1.0rc1 2022-03-17T19:07:04.1531742Z * [new tag] v1.1.0 -> v1.1.0 2022-03-17T19:07:04.1533010Z * [new tag] v1.1.0a0 -> v1.1.0a0 2022-03-17T19:07:04.1534511Z * [new tag] v1.10.0 -> v1.10.0 2022-03-17T19:07:04.1535784Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2022-03-17T19:07:04.1537121Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2022-03-17T19:07:04.1537798Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2022-03-17T19:07:04.1539494Z * [new tag] v1.10.1 -> v1.10.1 2022-03-17T19:07:04.1540073Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2022-03-17T19:07:04.1541181Z * [new tag] v1.10.2 -> v1.10.2 2022-03-17T19:07:04.1542406Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2022-03-17T19:07:04.1543579Z * [new tag] v1.11.0 -> v1.11.0 2022-03-17T19:07:04.1544860Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2022-03-17T19:07:04.1546142Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2022-03-17T19:07:04.1547354Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2022-03-17T19:07:04.1548586Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2022-03-17T19:07:04.1549787Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2022-03-17T19:07:04.1550539Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2022-03-17T19:07:04.1551437Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2022-03-17T19:07:04.1552731Z * [new tag] v1.2.0 -> v1.2.0 2022-03-17T19:07:04.1553899Z * [new tag] v1.2.0a0 -> v1.2.0a0 2022-03-17T19:07:04.1555006Z * [new tag] v1.3.0 -> v1.3.0 2022-03-17T19:07:04.1556150Z * [new tag] v1.3.0a0 -> v1.3.0a0 2022-03-17T19:07:04.1556950Z * [new tag] v1.3.1 -> v1.3.1 2022-03-17T19:07:04.1558229Z * [new tag] v1.4.0 -> v1.4.0 2022-03-17T19:07:04.1559442Z * [new tag] v1.4.0a0 -> v1.4.0a0 2022-03-17T19:07:04.1560528Z * [new tag] v1.4.1 -> v1.4.1 2022-03-17T19:07:04.1561584Z * [new tag] v1.5.0 -> v1.5.0 2022-03-17T19:07:04.1562840Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2022-03-17T19:07:04.1564028Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2022-03-17T19:07:04.1566328Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2022-03-17T19:07:04.1567277Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2022-03-17T19:07:04.1568069Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2022-03-17T19:07:04.1569387Z * [new tag] v1.5.1 -> v1.5.1 2022-03-17T19:07:04.1570275Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2022-03-17T19:07:04.1571473Z * [new tag] v1.6.0 -> v1.6.0 2022-03-17T19:07:04.1572550Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2022-03-17T19:07:04.1573800Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2022-03-17T19:07:04.1574973Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2022-03-17T19:07:04.1576160Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2022-03-17T19:07:04.1577386Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2022-03-17T19:07:04.1578567Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2022-03-17T19:07:04.1579357Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2022-03-17T19:07:04.1580636Z * [new tag] v1.7.0 -> v1.7.0 2022-03-17T19:07:04.1581964Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2022-03-17T19:07:04.1583207Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2022-03-17T19:07:04.1584482Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2022-03-17T19:07:04.1585348Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2022-03-17T19:07:04.1586931Z * [new tag] v1.7.1 -> v1.7.1 2022-03-17T19:07:04.1588148Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2022-03-17T19:07:04.1589407Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2022-03-17T19:07:04.1590156Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2022-03-17T19:07:04.1591378Z * [new tag] v1.8.0 -> v1.8.0 2022-03-17T19:07:04.1592416Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2022-03-17T19:07:04.1593905Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2022-03-17T19:07:04.1594967Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2022-03-17T19:07:04.1596108Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2022-03-17T19:07:04.1596890Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2022-03-17T19:07:04.1598013Z * [new tag] v1.8.1 -> v1.8.1 2022-03-17T19:07:04.1599228Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2022-03-17T19:07:04.1600175Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2022-03-17T19:07:04.1601208Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2022-03-17T19:07:04.1603344Z * [new tag] v1.8.2 -> v1.8.2 2022-03-17T19:07:04.1603746Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2022-03-17T19:07:04.1605389Z * [new tag] v1.9.0 -> v1.9.0 2022-03-17T19:07:04.1606318Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2022-03-17T19:07:04.1608214Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2022-03-17T19:07:04.1609266Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2022-03-17T19:07:04.1610060Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2022-03-17T19:07:04.1611469Z * [new tag] v1.9.1 -> v1.9.1 2022-03-17T19:07:04.1613375Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2022-03-17T19:07:04.1613812Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2022-03-17T19:07:04.2691877Z [command]/usr/bin/git rev-parse --verify --quiet 4b4f652f799c732a7280052e44a803a4cf6084de^{object} 2022-03-17T19:07:04.2717346Z 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:07:04.2721376Z ##[endgroup] 2022-03-17T19:07:04.2721750Z ##[group]Determining the checkout info 2022-03-17T19:07:04.2723332Z ##[endgroup] 2022-03-17T19:07:04.2723650Z ##[group]Checking out the ref 2022-03-17T19:07:04.2728032Z [command]/usr/bin/git checkout --progress --force 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:07:05.2428284Z Note: switching to '4b4f652f799c732a7280052e44a803a4cf6084de'. 2022-03-17T19:07:05.2428550Z 2022-03-17T19:07:05.2428845Z You are in 'detached HEAD' state. You can look around, make experimental 2022-03-17T19:07:05.2429248Z changes and commit them, and you can discard any commits you make in this 2022-03-17T19:07:05.2429658Z state without impacting any branches by switching back to a branch. 2022-03-17T19:07:05.2429891Z 2022-03-17T19:07:05.2430083Z If you want to create a new branch to retain commits you create, you may 2022-03-17T19:07:05.2430530Z do so (now or later) by using -c with the switch command. Example: 2022-03-17T19:07:05.2430749Z 2022-03-17T19:07:05.2430934Z git switch -c 2022-03-17T19:07:05.2431131Z 2022-03-17T19:07:05.2431260Z Or undo this operation with: 2022-03-17T19:07:05.2431419Z 2022-03-17T19:07:05.2431527Z git switch - 2022-03-17T19:07:05.2431682Z 2022-03-17T19:07:05.2431907Z Turn off this advice by setting config variable advice.detachedHead to false 2022-03-17T19:07:05.2432323Z 2022-03-17T19:07:05.2432494Z HEAD is now at 4b4f652f79 [3/5] Put JIT source inside flatbuffer (#74245) 2022-03-17T19:07:05.2478755Z ##[endgroup] 2022-03-17T19:07:05.2479404Z ##[group]Setting up auth for fetching submodules 2022-03-17T19:07:05.2494141Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4923cbfe-3c06-43a3-ad9c-d26db94b326f' before making global git config changes 2022-03-17T19:07:05.2498440Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2022-03-17T19:07:05.2550735Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2022-03-17T19:07:05.2578478Z [command]/usr/bin/git config --global url.https://github.com/.insteadOf git@github.com: 2022-03-17T19:07:05.2604180Z ##[endgroup] 2022-03-17T19:07:05.2604639Z ##[group]Fetching submodules 2022-03-17T19:07:05.2608927Z [command]/usr/bin/git submodule sync --recursive 2022-03-17T19:07:05.2924097Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2022-03-17T19:07:05.3227164Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2022-03-17T19:07:05.3229705Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2022-03-17T19:07:05.3232824Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2022-03-17T19:07:05.3236054Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2022-03-17T19:07:05.3239438Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK' 2022-03-17T19:07:05.3242929Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2022-03-17T19:07:05.3246553Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2022-03-17T19:07:05.3250316Z Submodule 'third_party/breakpad' (https://github.com/driazati/breakpad.git) registered for path 'third_party/breakpad' 2022-03-17T19:07:05.3254198Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2022-03-17T19:07:05.3258215Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub' 2022-03-17T19:07:05.3263027Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2022-03-17T19:07:05.3267292Z Submodule 'third_party/eigen' (https://github.com/eigenteam/eigen-git-mirror.git) registered for path 'third_party/eigen' 2022-03-17T19:07:05.3271680Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2022-03-17T19:07:05.3276394Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2022-03-17T19:07:05.3280839Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2022-03-17T19:07:05.3285682Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2022-03-17T19:07:05.3290588Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:07:05.3295781Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2022-03-17T19:07:05.3301207Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2022-03-17T19:07:05.3307417Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2022-03-17T19:07:05.3312794Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake' 2022-03-17T19:07:05.3318441Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2022-03-17T19:07:05.3324118Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2022-03-17T19:07:05.3330000Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse' 2022-03-17T19:07:05.3335876Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2022-03-17T19:07:05.3342630Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt' 2022-03-17T19:07:05.3348836Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2022-03-17T19:07:05.3355177Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2022-03-17T19:07:05.3361671Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2022-03-17T19:07:05.3368333Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2022-03-17T19:07:05.3375050Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2022-03-17T19:07:05.3382338Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum' 2022-03-17T19:07:05.3389422Z Submodule 'third_party/python-peachpy' (https://github.com/Maratyszcza/PeachPy.git) registered for path 'third_party/python-peachpy' 2022-03-17T19:07:05.3396565Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six' 2022-03-17T19:07:05.3403781Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2022-03-17T19:07:05.3411143Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb' 2022-03-17T19:07:05.3418772Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2022-03-17T19:07:05.3426916Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd' 2022-03-17T19:07:05.3487577Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2022-03-17T19:07:05.6390740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2022-03-17T19:07:06.0239047Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2022-03-17T19:07:06.2588529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2022-03-17T19:07:06.5364381Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'... 2022-03-17T19:07:06.8153082Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2022-03-17T19:07:10.4382488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2022-03-17T19:07:10.9036078Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/breakpad'... 2022-03-17T19:07:12.0604552Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2022-03-17T19:07:12.5895131Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'... 2022-03-17T19:07:13.8798666Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2022-03-17T19:07:17.7146290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2022-03-17T19:07:22.4295874Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2022-03-17T19:07:23.0262096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2022-03-17T19:07:24.6106147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2022-03-17T19:07:25.6599477Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2022-03-17T19:07:25.8554269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2022-03-17T19:07:26.3122021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2022-03-17T19:07:26.6526282Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2022-03-17T19:07:27.6709999Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2022-03-17T19:07:28.0172569Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'... 2022-03-17T19:07:28.2069198Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2022-03-17T19:07:30.1989207Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2022-03-17T19:07:30.5897061Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'... 2022-03-17T19:07:30.9558433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2022-03-17T19:07:32.8785501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'... 2022-03-17T19:07:33.3082426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2022-03-17T19:07:33.5129780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2022-03-17T19:07:38.3714525Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2022-03-17T19:07:38.5552402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2022-03-17T19:07:38.7850398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2022-03-17T19:07:40.1269990Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'... 2022-03-17T19:07:40.3253243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2022-03-17T19:07:40.6054991Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'... 2022-03-17T19:07:40.9082986Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2022-03-17T19:07:41.4314614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'... 2022-03-17T19:07:43.7862814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2022-03-17T19:07:44.4498393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'... 2022-03-17T19:07:46.6186674Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2022-03-17T19:07:46.6568158Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2022-03-17T19:07:46.6915180Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2022-03-17T19:07:46.7442431Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2022-03-17T19:07:46.7968288Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c' 2022-03-17T19:07:47.5477295Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba' 2022-03-17T19:07:47.5978909Z Submodule path 'third_party/benchmark': checked out 'e991355c02b93fe17713efe04cbc2e278e00fdbd' 2022-03-17T19:07:47.8154399Z Submodule path 'third_party/breakpad': checked out '7d188f679d4ae0a5bd06408a3047d69ef8eef848' 2022-03-17T19:07:47.8207570Z Submodule 'src/third_party/lss' (https://chromium.googlesource.com/linux-syscall-support) registered for path 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:07:47.8254506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/breakpad/src/third_party/lss'... 2022-03-17T19:07:48.0556740Z Submodule path 'third_party/breakpad/src/third_party/lss': checked out 'e1e7b0ad8ee99a875b272c8e33e308472e897660' 2022-03-17T19:07:48.2041612Z Submodule path 'third_party/cpuinfo': checked out '5916273f79a21551890fd3d56fc5375a78d1598d' 2022-03-17T19:07:48.2710699Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4' 2022-03-17T19:07:48.6497403Z Submodule path 'third_party/cudnn_frontend': checked out '7b83dba83fa31381aeca508d89aab94f4639ac6d' 2022-03-17T19:07:49.0400483Z Submodule path 'third_party/eigen': checked out 'd41dc4dd74acce21fb210e7625d5d135751fa9e5' 2022-03-17T19:07:49.1266459Z Submodule path 'third_party/fbgemm': checked out 'bedfcaeea242e7d900275d5650c744307d34ca03' 2022-03-17T19:07:49.1317028Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:07:49.1320007Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:07:49.1323214Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:07:49.1368263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2022-03-17T19:07:50.0199046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2022-03-17T19:07:50.5469435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2022-03-17T19:07:51.5970011Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out '8b35b4cffb62ecb58a903bf91cb7537d7a672211' 2022-03-17T19:07:51.7450149Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2022-03-17T19:07:51.8378004Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2022-03-17T19:07:51.9683000Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa' 2022-03-17T19:07:52.0335686Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2022-03-17T19:07:52.0690543Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2022-03-17T19:07:52.1411503Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2022-03-17T19:07:52.1943446Z Submodule path 'third_party/gloo': checked out 'c22a5cfba94edf8ea4f53a174d38aa0c629d070f' 2022-03-17T19:07:52.2747457Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2022-03-17T19:07:52.3126424Z Submodule path 'third_party/ideep': checked out '4a56ab2c3f61c44e0f8ea241beeb732b7d70dc5b' 2022-03-17T19:07:52.3174240Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2022-03-17T19:07:52.3216429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2022-03-17T19:07:57.7849202Z Submodule path 'third_party/ideep/mkl-dnn': checked out '16548c949f392376996ff180c695908ece5f83f0' 2022-03-17T19:07:57.7912122Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:07:57.7960043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'... 2022-03-17T19:08:03.1930937Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out 'a9302535553c73243c632ad3c4c80beec3d19a1e' 2022-03-17T19:08:03.2333618Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724' 2022-03-17T19:08:03.3635064Z Submodule path 'third_party/kineto': checked out '422591614ae4ad2cbc85ea24de48727e8ec12481' 2022-03-17T19:08:03.3685134Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:08:03.3688150Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:08:03.3732422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2022-03-17T19:08:04.4696220Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2022-03-17T19:08:05.4932596Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd' 2022-03-17T19:08:05.5810107Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2022-03-17T19:08:05.6297399Z Submodule path 'third_party/nccl/nccl': checked out '7e515921295adaab72adf56ea71a0fafb0ecb5f3' 2022-03-17T19:08:05.6706869Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a' 2022-03-17T19:08:05.9872151Z Submodule path 'third_party/onnx': checked out '96046b8ccfb8e6fa82f6b2b34b3d56add2e8849c' 2022-03-17T19:08:05.9937233Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2022-03-17T19:08:05.9940190Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2022-03-17T19:08:05.9998377Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2022-03-17T19:08:06.5085583Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2022-03-17T19:08:07.3967503Z Submodule path 'third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-03-17T19:08:07.4599539Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '59a2ac2745d8a57ac94c6accced73620d59fb844' 2022-03-17T19:08:07.5027424Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f' 2022-03-17T19:08:07.5076962Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:08:07.5120086Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'... 2022-03-17T19:08:09.5214589Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8' 2022-03-17T19:08:09.5279434Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:08:09.5282705Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:08:09.5333157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'... 2022-03-17T19:08:10.0038998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'... 2022-03-17T19:08:10.8952247Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508' 2022-03-17T19:08:10.9950736Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c' 2022-03-17T19:08:11.0009060Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:08:11.0053739Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'... 2022-03-17T19:08:11.2661968Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-03-17T19:08:11.3041814Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a' 2022-03-17T19:08:11.6424369Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2022-03-17T19:08:11.6481100Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:08:11.6484212Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2022-03-17T19:08:11.6533586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2022-03-17T19:08:12.0964691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2022-03-17T19:08:13.1236409Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2022-03-17T19:08:13.2303826Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2022-03-17T19:08:13.2670681Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2022-03-17T19:08:13.3050797Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413' 2022-03-17T19:08:13.3665155Z Submodule path 'third_party/pybind11': checked out '8de7772cc72daca8e947b79b83fea46214931604' 2022-03-17T19:08:13.4019487Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7' 2022-03-17T19:08:13.4621999Z Submodule path 'third_party/python-peachpy': checked out '07d8fde8ac45d7705129475c0f94ed8925b93473' 2022-03-17T19:08:13.4981041Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a' 2022-03-17T19:08:13.5754118Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff' 2022-03-17T19:08:13.7346145Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9' 2022-03-17T19:08:13.7899385Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2022-03-17T19:08:13.7948500Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:08:13.7951309Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:08:13.7954714Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:08:13.7957894Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:08:13.8003036Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2022-03-17T19:08:14.8361686Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2022-03-17T19:08:15.0844761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2022-03-17T19:08:16.2985680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2022-03-17T19:08:17.2472444Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2022-03-17T19:08:17.2910443Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2022-03-17T19:08:17.3964520Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2022-03-17T19:08:17.4559279Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2022-03-17T19:08:17.4616926Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:08:17.4660330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2022-03-17T19:08:17.7188274Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2022-03-17T19:08:17.9073148Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8' 2022-03-17T19:08:17.9164209Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2022-03-17T19:08:17.9483252Z Entering 'android/libs/fbjni' 2022-03-17T19:08:17.9524221Z Entering 'third_party/FP16' 2022-03-17T19:08:17.9566082Z Entering 'third_party/FXdiv' 2022-03-17T19:08:17.9606142Z Entering 'third_party/NNPACK' 2022-03-17T19:08:17.9646876Z Entering 'third_party/QNNPACK' 2022-03-17T19:08:17.9688200Z Entering 'third_party/XNNPACK' 2022-03-17T19:08:17.9739944Z Entering 'third_party/benchmark' 2022-03-17T19:08:17.9780485Z Entering 'third_party/breakpad' 2022-03-17T19:08:17.9822951Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:08:17.9864377Z Entering 'third_party/cpuinfo' 2022-03-17T19:08:17.9904310Z Entering 'third_party/cub' 2022-03-17T19:08:17.9944300Z Entering 'third_party/cudnn_frontend' 2022-03-17T19:08:17.9990534Z Entering 'third_party/eigen' 2022-03-17T19:08:18.0032770Z Entering 'third_party/fbgemm' 2022-03-17T19:08:18.0072392Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:08:18.0113076Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:08:18.0153458Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:08:18.0195928Z Entering 'third_party/flatbuffers' 2022-03-17T19:08:18.0239395Z Entering 'third_party/fmt' 2022-03-17T19:08:18.0279983Z Entering 'third_party/foxi' 2022-03-17T19:08:18.0320294Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:08:18.0360385Z Entering 'third_party/gloo' 2022-03-17T19:08:18.0401586Z Entering 'third_party/googletest' 2022-03-17T19:08:18.0441559Z Entering 'third_party/ideep' 2022-03-17T19:08:18.0481509Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T19:08:18.0522905Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:08:18.0569820Z Entering 'third_party/ios-cmake' 2022-03-17T19:08:18.0609209Z Entering 'third_party/kineto' 2022-03-17T19:08:18.0649907Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:08:18.0689827Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:08:18.0730781Z Entering 'third_party/nccl/nccl' 2022-03-17T19:08:18.0770551Z Entering 'third_party/neon2sse' 2022-03-17T19:08:18.0812456Z Entering 'third_party/onnx' 2022-03-17T19:08:18.0864726Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T19:08:18.0904653Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T19:08:18.0947299Z Entering 'third_party/onnx-tensorrt' 2022-03-17T19:08:18.0985990Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:08:18.1030425Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:08:18.1069941Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:08:18.1109419Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:08:18.1152848Z Entering 'third_party/pocketfft' 2022-03-17T19:08:18.1193063Z Entering 'third_party/protobuf' 2022-03-17T19:08:18.1237238Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:08:18.1276958Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T19:08:18.1319634Z Entering 'third_party/psimd' 2022-03-17T19:08:18.1360749Z Entering 'third_party/pthreadpool' 2022-03-17T19:08:18.1401214Z Entering 'third_party/pybind11' 2022-03-17T19:08:18.1441527Z Entering 'third_party/python-enum' 2022-03-17T19:08:18.1482466Z Entering 'third_party/python-peachpy' 2022-03-17T19:08:18.1522569Z Entering 'third_party/python-six' 2022-03-17T19:08:18.1563110Z Entering 'third_party/sleef' 2022-03-17T19:08:18.1604398Z Entering 'third_party/tbb' 2022-03-17T19:08:18.1647849Z Entering 'third_party/tensorpipe' 2022-03-17T19:08:18.1687660Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:08:18.1728097Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:08:18.1767778Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:08:18.1807403Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:08:18.1846899Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:08:18.1889123Z Entering 'third_party/zstd' 2022-03-17T19:08:18.1941422Z ##[endgroup] 2022-03-17T19:08:18.1942094Z ##[group]Persisting credentials for submodules 2022-03-17T19:08:18.1949727Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || : 2022-03-17T19:08:18.2265790Z Entering 'android/libs/fbjni' 2022-03-17T19:08:18.2305441Z Entering 'third_party/FP16' 2022-03-17T19:08:18.2345802Z Entering 'third_party/FXdiv' 2022-03-17T19:08:18.2385529Z Entering 'third_party/NNPACK' 2022-03-17T19:08:18.2426261Z Entering 'third_party/QNNPACK' 2022-03-17T19:08:18.2468817Z Entering 'third_party/XNNPACK' 2022-03-17T19:08:18.2520836Z Entering 'third_party/benchmark' 2022-03-17T19:08:18.2560510Z Entering 'third_party/breakpad' 2022-03-17T19:08:18.2603233Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:08:18.2644822Z Entering 'third_party/cpuinfo' 2022-03-17T19:08:18.2685257Z Entering 'third_party/cub' 2022-03-17T19:08:18.2725890Z Entering 'third_party/cudnn_frontend' 2022-03-17T19:08:18.2771872Z Entering 'third_party/eigen' 2022-03-17T19:08:18.2814869Z Entering 'third_party/fbgemm' 2022-03-17T19:08:18.2853934Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:08:18.2895773Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:08:18.2936038Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:08:18.2979433Z Entering 'third_party/flatbuffers' 2022-03-17T19:08:18.3021264Z Entering 'third_party/fmt' 2022-03-17T19:08:18.3060461Z Entering 'third_party/foxi' 2022-03-17T19:08:18.3099728Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:08:18.3138965Z Entering 'third_party/gloo' 2022-03-17T19:08:18.3178846Z Entering 'third_party/googletest' 2022-03-17T19:08:18.3218365Z Entering 'third_party/ideep' 2022-03-17T19:08:18.3258092Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T19:08:18.3298407Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:08:18.3345413Z Entering 'third_party/ios-cmake' 2022-03-17T19:08:18.3385044Z Entering 'third_party/kineto' 2022-03-17T19:08:18.3423771Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:08:18.3463215Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:08:18.3505429Z Entering 'third_party/nccl/nccl' 2022-03-17T19:08:18.3544612Z Entering 'third_party/neon2sse' 2022-03-17T19:08:18.3584795Z Entering 'third_party/onnx' 2022-03-17T19:08:18.3636760Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T19:08:18.3677527Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T19:08:18.3719670Z Entering 'third_party/onnx-tensorrt' 2022-03-17T19:08:18.3758136Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:08:18.3803015Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:08:18.3844076Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:08:18.3883164Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:08:18.3926703Z Entering 'third_party/pocketfft' 2022-03-17T19:08:18.3965375Z Entering 'third_party/protobuf' 2022-03-17T19:08:18.4007255Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:08:18.4046442Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T19:08:18.4087123Z Entering 'third_party/psimd' 2022-03-17T19:08:18.4126541Z Entering 'third_party/pthreadpool' 2022-03-17T19:08:18.4165288Z Entering 'third_party/pybind11' 2022-03-17T19:08:18.4205270Z Entering 'third_party/python-enum' 2022-03-17T19:08:18.4244075Z Entering 'third_party/python-peachpy' 2022-03-17T19:08:18.4283339Z Entering 'third_party/python-six' 2022-03-17T19:08:18.4324099Z Entering 'third_party/sleef' 2022-03-17T19:08:18.4362806Z Entering 'third_party/tbb' 2022-03-17T19:08:18.4404453Z Entering 'third_party/tensorpipe' 2022-03-17T19:08:18.4443055Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:08:18.4482972Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:08:18.4521565Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:08:18.4560322Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:08:18.4598223Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:08:18.4639056Z Entering 'third_party/zstd' 2022-03-17T19:08:18.4692269Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url 2022-03-17T19:08:18.5006407Z Entering 'android/libs/fbjni' 2022-03-17T19:08:18.5044405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2022-03-17T19:08:18.5061251Z Entering 'third_party/FP16' 2022-03-17T19:08:18.5099298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2022-03-17T19:08:18.5116514Z Entering 'third_party/FXdiv' 2022-03-17T19:08:18.5153645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2022-03-17T19:08:18.5170701Z Entering 'third_party/NNPACK' 2022-03-17T19:08:18.5208844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2022-03-17T19:08:18.5226387Z Entering 'third_party/QNNPACK' 2022-03-17T19:08:18.5263646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config remote.origin.url 2022-03-17T19:08:18.5280660Z Entering 'third_party/XNNPACK' 2022-03-17T19:08:18.5317500Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2022-03-17T19:08:18.5346048Z Entering 'third_party/benchmark' 2022-03-17T19:08:18.5383028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2022-03-17T19:08:18.5400065Z Entering 'third_party/breakpad' 2022-03-17T19:08:18.5437170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/breakpad/config remote.origin.url 2022-03-17T19:08:18.5454911Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:08:18.5491716Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/breakpad/modules/src/third_party/lss/config remote.origin.url 2022-03-17T19:08:18.5510416Z Entering 'third_party/cpuinfo' 2022-03-17T19:08:18.5547431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2022-03-17T19:08:18.5564776Z Entering 'third_party/cub' 2022-03-17T19:08:18.5602469Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config remote.origin.url 2022-03-17T19:08:18.5619508Z Entering 'third_party/cudnn_frontend' 2022-03-17T19:08:18.5656896Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2022-03-17T19:08:18.5680384Z Entering 'third_party/eigen' 2022-03-17T19:08:18.5718118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2022-03-17T19:08:18.5737557Z Entering 'third_party/fbgemm' 2022-03-17T19:08:18.5775576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2022-03-17T19:08:18.5792754Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:08:18.5829421Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2022-03-17T19:08:18.5845977Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:08:18.5883412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2022-03-17T19:08:18.5900453Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:08:18.5936634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2022-03-17T19:08:18.5954563Z Entering 'third_party/flatbuffers' 2022-03-17T19:08:18.5991405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2022-03-17T19:08:18.6010151Z Entering 'third_party/fmt' 2022-03-17T19:08:18.6048123Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2022-03-17T19:08:18.6065473Z Entering 'third_party/foxi' 2022-03-17T19:08:18.6103538Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2022-03-17T19:08:18.6120053Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:08:18.6156787Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2022-03-17T19:08:18.6173659Z Entering 'third_party/gloo' 2022-03-17T19:08:18.6210489Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2022-03-17T19:08:18.6228743Z Entering 'third_party/googletest' 2022-03-17T19:08:18.6264915Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2022-03-17T19:08:18.6281480Z Entering 'third_party/ideep' 2022-03-17T19:08:18.6318912Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2022-03-17T19:08:18.6334787Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T19:08:18.6371731Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2022-03-17T19:08:18.6390192Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:08:18.6426416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config remote.origin.url 2022-03-17T19:08:18.6449139Z Entering 'third_party/ios-cmake' 2022-03-17T19:08:18.6485396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config remote.origin.url 2022-03-17T19:08:18.6502350Z Entering 'third_party/kineto' 2022-03-17T19:08:18.6538566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2022-03-17T19:08:18.6555221Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:08:18.6591257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2022-03-17T19:08:18.6607978Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:08:18.6644715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2022-03-17T19:08:18.6664099Z Entering 'third_party/nccl/nccl' 2022-03-17T19:08:18.6700956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2022-03-17T19:08:18.6717690Z Entering 'third_party/neon2sse' 2022-03-17T19:08:18.6754729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config remote.origin.url 2022-03-17T19:08:18.6771285Z Entering 'third_party/onnx' 2022-03-17T19:08:18.6807793Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2022-03-17T19:08:18.6837443Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T19:08:18.6873497Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-03-17T19:08:18.6890509Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T19:08:18.6927426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-03-17T19:08:18.6946307Z Entering 'third_party/onnx-tensorrt' 2022-03-17T19:08:18.6982873Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config remote.origin.url 2022-03-17T19:08:18.6998156Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:08:18.7034328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config remote.origin.url 2022-03-17T19:08:18.7055553Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:08:18.7092545Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2022-03-17T19:08:18.7109396Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:08:18.7146871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2022-03-17T19:08:18.7162915Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:08:18.7200182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-03-17T19:08:18.7222186Z Entering 'third_party/pocketfft' 2022-03-17T19:08:18.7259134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2022-03-17T19:08:18.7275741Z Entering 'third_party/protobuf' 2022-03-17T19:08:18.7314384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2022-03-17T19:08:18.7333811Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:08:18.7371377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2022-03-17T19:08:18.7389144Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T19:08:18.7425723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2022-03-17T19:08:18.7444123Z Entering 'third_party/psimd' 2022-03-17T19:08:18.7481731Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2022-03-17T19:08:18.7498369Z Entering 'third_party/pthreadpool' 2022-03-17T19:08:18.7535366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2022-03-17T19:08:18.7552220Z Entering 'third_party/pybind11' 2022-03-17T19:08:18.7589051Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2022-03-17T19:08:18.7606042Z Entering 'third_party/python-enum' 2022-03-17T19:08:18.7643809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config remote.origin.url 2022-03-17T19:08:18.7660341Z Entering 'third_party/python-peachpy' 2022-03-17T19:08:18.7696768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2022-03-17T19:08:18.7714057Z Entering 'third_party/python-six' 2022-03-17T19:08:18.7751295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config remote.origin.url 2022-03-17T19:08:18.7767679Z Entering 'third_party/sleef' 2022-03-17T19:08:18.7803936Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2022-03-17T19:08:18.7822012Z Entering 'third_party/tbb' 2022-03-17T19:08:18.7858260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config remote.origin.url 2022-03-17T19:08:18.7877653Z Entering 'third_party/tensorpipe' 2022-03-17T19:08:18.7914649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2022-03-17T19:08:18.7931389Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:08:18.7968118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2022-03-17T19:08:18.7985855Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:08:18.8022897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2022-03-17T19:08:18.8039406Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:08:18.8077199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2022-03-17T19:08:18.8094931Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:08:18.8131079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2022-03-17T19:08:18.8147465Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:08:18.8184660Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2022-03-17T19:08:18.8204123Z Entering 'third_party/zstd' 2022-03-17T19:08:18.8241779Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config remote.origin.url 2022-03-17T19:08:18.9055647Z [command]/usr/bin/git submodule foreach --recursive git config --local 'url.https://github.com/.insteadOf' 'git@github.com:' 2022-03-17T19:08:18.9370396Z Entering 'android/libs/fbjni' 2022-03-17T19:08:18.9411453Z Entering 'third_party/FP16' 2022-03-17T19:08:18.9453928Z Entering 'third_party/FXdiv' 2022-03-17T19:08:18.9494434Z Entering 'third_party/NNPACK' 2022-03-17T19:08:18.9536524Z Entering 'third_party/QNNPACK' 2022-03-17T19:08:18.9577167Z Entering 'third_party/XNNPACK' 2022-03-17T19:08:18.9630482Z Entering 'third_party/benchmark' 2022-03-17T19:08:18.9670707Z Entering 'third_party/breakpad' 2022-03-17T19:08:18.9712611Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T19:08:18.9755506Z Entering 'third_party/cpuinfo' 2022-03-17T19:08:18.9796067Z Entering 'third_party/cub' 2022-03-17T19:08:18.9836799Z Entering 'third_party/cudnn_frontend' 2022-03-17T19:08:18.9884626Z Entering 'third_party/eigen' 2022-03-17T19:08:18.9928817Z Entering 'third_party/fbgemm' 2022-03-17T19:08:18.9969185Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T19:08:19.0009562Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T19:08:19.0049999Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T19:08:19.0091748Z Entering 'third_party/flatbuffers' 2022-03-17T19:08:19.0135986Z Entering 'third_party/fmt' 2022-03-17T19:08:19.0177208Z Entering 'third_party/foxi' 2022-03-17T19:08:19.0217265Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T19:08:19.0258116Z Entering 'third_party/gloo' 2022-03-17T19:08:19.0298961Z Entering 'third_party/googletest' 2022-03-17T19:08:19.0339741Z Entering 'third_party/ideep' 2022-03-17T19:08:19.0379229Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T19:08:19.0422228Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T19:08:19.0470079Z Entering 'third_party/ios-cmake' 2022-03-17T19:08:19.0510831Z Entering 'third_party/kineto' 2022-03-17T19:08:19.0551454Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T19:08:19.0592162Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T19:08:19.0633594Z Entering 'third_party/nccl/nccl' 2022-03-17T19:08:19.0674183Z Entering 'third_party/neon2sse' 2022-03-17T19:08:19.0713582Z Entering 'third_party/onnx' 2022-03-17T19:08:19.0766236Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T19:08:19.0808208Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T19:08:19.0850799Z Entering 'third_party/onnx-tensorrt' 2022-03-17T19:08:19.0890819Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T19:08:19.0935796Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T19:08:19.0977897Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T19:08:19.1018239Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T19:08:19.1064298Z Entering 'third_party/pocketfft' 2022-03-17T19:08:19.1104377Z Entering 'third_party/protobuf' 2022-03-17T19:08:19.1148777Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T19:08:19.1189490Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T19:08:19.1232304Z Entering 'third_party/psimd' 2022-03-17T19:08:19.1273183Z Entering 'third_party/pthreadpool' 2022-03-17T19:08:19.1314171Z Entering 'third_party/pybind11' 2022-03-17T19:08:19.1354405Z Entering 'third_party/python-enum' 2022-03-17T19:08:19.1395298Z Entering 'third_party/python-peachpy' 2022-03-17T19:08:19.1436367Z Entering 'third_party/python-six' 2022-03-17T19:08:19.1477299Z Entering 'third_party/sleef' 2022-03-17T19:08:19.1518702Z Entering 'third_party/tbb' 2022-03-17T19:08:19.1562279Z Entering 'third_party/tensorpipe' 2022-03-17T19:08:19.1602424Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T19:08:19.1642599Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T19:08:19.1682292Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T19:08:19.1722840Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T19:08:19.1761542Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T19:08:19.1804138Z Entering 'third_party/zstd' 2022-03-17T19:08:19.1854492Z ##[endgroup] 2022-03-17T19:08:19.1935071Z [command]/usr/bin/git log -1 --format='%H' 2022-03-17T19:08:19.1971387Z '4b4f652f799c732a7280052e44a803a4cf6084de' 2022-03-17T19:08:19.2041376Z ##[group]Run # Remove any artifacts from the previous checkouts 2022-03-17T19:08:19.2041799Z # Remove any artifacts from the previous checkouts 2022-03-17T19:08:19.2042108Z git clean -fxd 2022-03-17T19:08:19.2054281Z shell: /usr/bin/bash -e {0} 2022-03-17T19:08:19.2054530Z env: 2022-03-17T19:08:19.2054823Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:08:19.2055331Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:08:19.2055809Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:08:19.2056230Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:08:19.2056591Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:08:19.2056941Z IN_CI: 1 2022-03-17T19:08:19.2057163Z IS_GHA: 1 2022-03-17T19:08:19.2057377Z IN_WHEEL_TEST: 1 2022-03-17T19:08:19.2057680Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:08:19.2058098Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:08:19.2058410Z PR_LABELS: [] 2022-03-17T19:08:19.2058917Z GITHUB_TOKEN: *** 2022-03-17T19:08:19.2059196Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:08:19.2059439Z PR_NUMBER: 2022-03-17T19:08:19.2059710Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:08:19.2059985Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:08:19.2060543Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:08:19.2061069Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:08:19.2061604Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:08:19.2061926Z TEST_CONFIG: distributed 2022-03-17T19:08:19.2062168Z SHARD_NUMBER: 1 2022-03-17T19:08:19.2062411Z NUM_TEST_SHARDS: 1 2022-03-17T19:08:19.2062646Z PR_BODY: 2022-03-17T19:08:19.2062862Z ##[endgroup] 2022-03-17T19:08:19.2241683Z ##[group]Run retry () { 2022-03-17T19:08:19.2242010Z retry () { 2022-03-17T19:08:19.2242273Z  "$@" || (sleep 1 && "$@") || (sleep 2 && "$@") 2022-03-17T19:08:19.2242536Z } 2022-03-17T19:08:19.2242810Z retry docker pull "${DOCKER_IMAGE}" 2022-03-17T19:08:19.2254359Z shell: /usr/bin/bash -e {0} 2022-03-17T19:08:19.2254615Z env: 2022-03-17T19:08:19.2254905Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:08:19.2255416Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:08:19.2255895Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:08:19.2256319Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:08:19.2256689Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:08:19.2256917Z IN_CI: 1 2022-03-17T19:08:19.2257134Z IS_GHA: 1 2022-03-17T19:08:19.2257347Z IN_WHEEL_TEST: 1 2022-03-17T19:08:19.2257683Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:08:19.2258057Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:08:19.2258382Z PR_LABELS: [] 2022-03-17T19:08:19.2258809Z GITHUB_TOKEN: *** 2022-03-17T19:08:19.2259062Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:08:19.2259324Z PR_NUMBER: 2022-03-17T19:08:19.2259580Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:08:19.2259866Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:08:19.2260123Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:08:19.2260940Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:08:19.2261500Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:08:19.2261812Z TEST_CONFIG: distributed 2022-03-17T19:08:19.2262064Z SHARD_NUMBER: 1 2022-03-17T19:08:19.2262287Z NUM_TEST_SHARDS: 1 2022-03-17T19:08:19.2262530Z PR_BODY: 2022-03-17T19:08:19.2262761Z ##[endgroup] 2022-03-17T19:08:19.4908000Z c3998772049b8a8ba52cedfa525c9feda04cb2c7: Pulling from pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:08:19.4908499Z 58690f9b18fc: Pulling fs layer 2022-03-17T19:08:19.4908760Z b51569e7c507: Pulling fs layer 2022-03-17T19:08:19.4909034Z da8ef40b9eca: Pulling fs layer 2022-03-17T19:08:19.4909312Z fb15d46c38dc: Pulling fs layer 2022-03-17T19:08:19.4909865Z 4394b9f66b9a: Pulling fs layer 2022-03-17T19:08:19.4910192Z 22c97c628c7c: Pulling fs layer 2022-03-17T19:08:19.4910598Z 4603e084f865: Pulling fs layer 2022-03-17T19:08:19.4911043Z 0efe41560fc9: Pulling fs layer 2022-03-17T19:08:19.4911585Z 2a708e46e4ce: Pulling fs layer 2022-03-17T19:08:19.4929896Z a88643889c30: Pulling fs layer 2022-03-17T19:08:19.4930336Z fb15d46c38dc: Waiting 2022-03-17T19:08:19.4930796Z 0a97b47d3470: Pulling fs layer 2022-03-17T19:08:19.4931051Z 95bff152db9a: Pulling fs layer 2022-03-17T19:08:19.4931309Z 4603e084f865: Waiting 2022-03-17T19:08:19.4931536Z 0efe41560fc9: Waiting 2022-03-17T19:08:19.4936298Z 22c97c628c7c: Waiting 2022-03-17T19:08:19.4936845Z 2a708e46e4ce: Waiting 2022-03-17T19:08:19.4937278Z a88643889c30: Waiting 2022-03-17T19:08:19.4937528Z f415fd767169: Pulling fs layer 2022-03-17T19:08:19.4937786Z 0a97b47d3470: Waiting 2022-03-17T19:08:19.4937955Z 2350347ab300: Pulling fs layer 2022-03-17T19:08:19.4938207Z 95bff152db9a: Waiting 2022-03-17T19:08:19.4938431Z f415fd767169: Waiting 2022-03-17T19:08:19.4938715Z 9ab0eac18dff: Pulling fs layer 2022-03-17T19:08:19.4938984Z 868da44a8860: Pulling fs layer 2022-03-17T19:08:19.4939368Z 9ab0eac18dff: Waiting 2022-03-17T19:08:19.4939822Z 2350347ab300: Waiting 2022-03-17T19:08:19.4940185Z cc0029abc03f: Pulling fs layer 2022-03-17T19:08:19.4941077Z c45e6042be05: Pulling fs layer 2022-03-17T19:08:19.4941564Z d1e7e8aafebd: Pulling fs layer 2022-03-17T19:08:19.4942118Z 868da44a8860: Waiting 2022-03-17T19:08:19.4942518Z bff23355a96d: Pulling fs layer 2022-03-17T19:08:19.4943094Z 4394b9f66b9a: Waiting 2022-03-17T19:08:19.4943352Z cc0029abc03f: Waiting 2022-03-17T19:08:19.4943589Z 8d733210f9f6: Pulling fs layer 2022-03-17T19:08:19.4943853Z e29500059cfe: Pulling fs layer 2022-03-17T19:08:19.4944090Z a7162e106409: Pulling fs layer 2022-03-17T19:08:19.4944339Z 156ef8f3a13c: Pulling fs layer 2022-03-17T19:08:19.4944591Z c45e6042be05: Waiting 2022-03-17T19:08:19.4944807Z e29500059cfe: Waiting 2022-03-17T19:08:19.4945069Z 08ba1108f7b7: Pulling fs layer 2022-03-17T19:08:19.4945313Z b1ca60ec900e: Pulling fs layer 2022-03-17T19:08:19.4945582Z 8d733210f9f6: Waiting 2022-03-17T19:08:19.4945797Z 9422c548b247: Pulling fs layer 2022-03-17T19:08:19.4946046Z a7162e106409: Waiting 2022-03-17T19:08:19.4946277Z 156ef8f3a13c: Waiting 2022-03-17T19:08:19.4946529Z b1ca60ec900e: Waiting 2022-03-17T19:08:19.4946776Z 2629d28d8057: Pulling fs layer 2022-03-17T19:08:19.4947019Z 08ba1108f7b7: Waiting 2022-03-17T19:08:19.4947256Z 9422c548b247: Waiting 2022-03-17T19:08:19.4947488Z 51b3d0db68b0: Pulling fs layer 2022-03-17T19:08:19.4947754Z df925f176b66: Pulling fs layer 2022-03-17T19:08:19.4948011Z 51b3d0db68b0: Waiting 2022-03-17T19:08:19.4948264Z 2629d28d8057: Waiting 2022-03-17T19:08:19.4948720Z 94e62fb0708b: Pulling fs layer 2022-03-17T19:08:19.4949176Z 2783d99ced8a: Pulling fs layer 2022-03-17T19:08:19.4949522Z d7107983cd6c: Pulling fs layer 2022-03-17T19:08:19.4950041Z 9b4350e11a74: Pulling fs layer 2022-03-17T19:08:19.4950502Z df925f176b66: Waiting 2022-03-17T19:08:19.4950947Z 40231bcc3933: Pulling fs layer 2022-03-17T19:08:19.4951375Z d7107983cd6c: Waiting 2022-03-17T19:08:19.4951703Z 12611fd1614b: Pulling fs layer 2022-03-17T19:08:19.4951935Z 40231bcc3933: Waiting 2022-03-17T19:08:19.4952176Z 2783d99ced8a: Waiting 2022-03-17T19:08:19.4952316Z 9b4350e11a74: Waiting 2022-03-17T19:08:19.4952571Z a8b79af452cb: Pulling fs layer 2022-03-17T19:08:19.4952818Z 94e62fb0708b: Waiting 2022-03-17T19:08:19.4953071Z 17f77bcc71e7: Pulling fs layer 2022-03-17T19:08:19.4953341Z a8b79af452cb: Waiting 2022-03-17T19:08:19.4953565Z 12611fd1614b: Waiting 2022-03-17T19:08:19.4953814Z 88ad1b55251d: Pulling fs layer 2022-03-17T19:08:19.4954065Z b8e74acbf1bb: Pulling fs layer 2022-03-17T19:08:19.4954319Z 17f77bcc71e7: Waiting 2022-03-17T19:08:19.4954556Z 88ad1b55251d: Waiting 2022-03-17T19:08:19.4954786Z 44810df15216: Pulling fs layer 2022-03-17T19:08:19.4955053Z 4af02869d275: Pulling fs layer 2022-03-17T19:08:19.4955290Z a23a0a601a32: Pulling fs layer 2022-03-17T19:08:19.4955552Z 45d5316499db: Pulling fs layer 2022-03-17T19:08:19.4955798Z cf097a682947: Pulling fs layer 2022-03-17T19:08:19.4956040Z 4af02869d275: Waiting 2022-03-17T19:08:19.4956261Z b8e74acbf1bb: Waiting 2022-03-17T19:08:19.4956499Z 44810df15216: Waiting 2022-03-17T19:08:19.4956730Z 45d5316499db: Waiting 2022-03-17T19:08:19.4956957Z 795311cb799f: Pulling fs layer 2022-03-17T19:08:19.4957212Z 575631837e37: Pulling fs layer 2022-03-17T19:08:19.4957592Z d9832f16d0b0: Pulling fs layer 2022-03-17T19:08:19.4957840Z cf097a682947: Waiting 2022-03-17T19:08:19.4958056Z 795311cb799f: Waiting 2022-03-17T19:08:19.5625479Z da8ef40b9eca: Verifying Checksum 2022-03-17T19:08:19.5625845Z da8ef40b9eca: Download complete 2022-03-17T19:08:19.5700062Z b51569e7c507: Download complete 2022-03-17T19:08:19.6392647Z fb15d46c38dc: Verifying Checksum 2022-03-17T19:08:19.6392928Z fb15d46c38dc: Download complete 2022-03-17T19:08:19.7008566Z 4394b9f66b9a: Verifying Checksum 2022-03-17T19:08:19.7008829Z 4394b9f66b9a: Download complete 2022-03-17T19:08:19.7709296Z 4603e084f865: Verifying Checksum 2022-03-17T19:08:19.7709594Z 4603e084f865: Download complete 2022-03-17T19:08:19.8033350Z 22c97c628c7c: Verifying Checksum 2022-03-17T19:08:19.8033621Z 22c97c628c7c: Download complete 2022-03-17T19:08:19.8372973Z 0efe41560fc9: Verifying Checksum 2022-03-17T19:08:19.8373833Z 0efe41560fc9: Download complete 2022-03-17T19:08:19.9185226Z a88643889c30: Verifying Checksum 2022-03-17T19:08:19.9185595Z a88643889c30: Download complete 2022-03-17T19:08:20.0077011Z 58690f9b18fc: Verifying Checksum 2022-03-17T19:08:20.0077561Z 58690f9b18fc: Download complete 2022-03-17T19:08:20.0917397Z 95bff152db9a: Verifying Checksum 2022-03-17T19:08:20.0917696Z 95bff152db9a: Download complete 2022-03-17T19:08:23.1193687Z 58690f9b18fc: Pull complete 2022-03-17T19:08:23.2592329Z b51569e7c507: Pull complete 2022-03-17T19:08:23.3959987Z da8ef40b9eca: Pull complete 2022-03-17T19:08:23.5154880Z fb15d46c38dc: Pull complete 2022-03-17T19:08:23.8190318Z 4394b9f66b9a: Pull complete 2022-03-17T19:08:24.1766256Z 22c97c628c7c: Pull complete 2022-03-17T19:08:24.3328607Z 4603e084f865: Pull complete 2022-03-17T19:08:24.4627042Z 0efe41560fc9: Pull complete 2022-03-17T19:08:28.5381877Z 2a708e46e4ce: Verifying Checksum 2022-03-17T19:08:28.5382402Z 2a708e46e4ce: Download complete 2022-03-17T19:08:28.6219253Z 2350347ab300: Verifying Checksum 2022-03-17T19:08:28.6219869Z 2350347ab300: Download complete 2022-03-17T19:08:30.3519654Z 9ab0eac18dff: Verifying Checksum 2022-03-17T19:08:30.3520016Z 9ab0eac18dff: Download complete 2022-03-17T19:08:30.4254895Z 868da44a8860: Verifying Checksum 2022-03-17T19:08:30.4255202Z 868da44a8860: Download complete 2022-03-17T19:08:30.4980947Z cc0029abc03f: Download complete 2022-03-17T19:08:30.5368809Z 0a97b47d3470: Verifying Checksum 2022-03-17T19:08:30.5369102Z 0a97b47d3470: Download complete 2022-03-17T19:08:30.5775836Z c45e6042be05: Verifying Checksum 2022-03-17T19:08:30.5776057Z c45e6042be05: Download complete 2022-03-17T19:08:30.6503311Z bff23355a96d: Download complete 2022-03-17T19:08:30.9753595Z d1e7e8aafebd: Verifying Checksum 2022-03-17T19:08:30.9754241Z d1e7e8aafebd: Download complete 2022-03-17T19:08:31.0390114Z e29500059cfe: Verifying Checksum 2022-03-17T19:08:31.0390705Z e29500059cfe: Download complete 2022-03-17T19:08:31.5605474Z a7162e106409: Verifying Checksum 2022-03-17T19:08:31.5605745Z a7162e106409: Download complete 2022-03-17T19:08:31.6322499Z 156ef8f3a13c: Verifying Checksum 2022-03-17T19:08:31.6322982Z 156ef8f3a13c: Download complete 2022-03-17T19:08:31.7073665Z 08ba1108f7b7: Verifying Checksum 2022-03-17T19:08:31.7074044Z 08ba1108f7b7: Download complete 2022-03-17T19:08:31.7763804Z b1ca60ec900e: Verifying Checksum 2022-03-17T19:08:31.7764239Z b1ca60ec900e: Download complete 2022-03-17T19:08:32.7251391Z 9422c548b247: Verifying Checksum 2022-03-17T19:08:32.7251741Z 9422c548b247: Download complete 2022-03-17T19:08:32.8216224Z 2629d28d8057: Verifying Checksum 2022-03-17T19:08:32.8216586Z 2629d28d8057: Download complete 2022-03-17T19:08:32.8954823Z 51b3d0db68b0: Verifying Checksum 2022-03-17T19:08:32.8955207Z 51b3d0db68b0: Download complete 2022-03-17T19:08:32.9895740Z df925f176b66: Verifying Checksum 2022-03-17T19:08:32.9896311Z df925f176b66: Download complete 2022-03-17T19:08:33.0713426Z 94e62fb0708b: Verifying Checksum 2022-03-17T19:08:33.0714008Z 94e62fb0708b: Download complete 2022-03-17T19:08:33.1503327Z 2783d99ced8a: Download complete 2022-03-17T19:08:34.1755053Z d7107983cd6c: Verifying Checksum 2022-03-17T19:08:34.1755629Z d7107983cd6c: Download complete 2022-03-17T19:08:34.2636906Z 9b4350e11a74: Download complete 2022-03-17T19:08:34.3547500Z 40231bcc3933: Verifying Checksum 2022-03-17T19:08:34.3547838Z 40231bcc3933: Download complete 2022-03-17T19:08:34.6056915Z 12611fd1614b: Verifying Checksum 2022-03-17T19:08:34.6057325Z 12611fd1614b: Download complete 2022-03-17T19:08:34.7282357Z a8b79af452cb: Verifying Checksum 2022-03-17T19:08:34.7282756Z a8b79af452cb: Download complete 2022-03-17T19:08:35.1380816Z 17f77bcc71e7: Verifying Checksum 2022-03-17T19:08:35.1381168Z 17f77bcc71e7: Download complete 2022-03-17T19:08:35.2169069Z 88ad1b55251d: Verifying Checksum 2022-03-17T19:08:35.2169442Z 88ad1b55251d: Download complete 2022-03-17T19:08:35.3096978Z b8e74acbf1bb: Download complete 2022-03-17T19:08:38.1826371Z 44810df15216: Verifying Checksum 2022-03-17T19:08:38.1826694Z 44810df15216: Download complete 2022-03-17T19:08:38.2622311Z 4af02869d275: Verifying Checksum 2022-03-17T19:08:38.2622655Z 4af02869d275: Download complete 2022-03-17T19:08:38.3612250Z a23a0a601a32: Verifying Checksum 2022-03-17T19:08:38.3612570Z a23a0a601a32: Download complete 2022-03-17T19:08:38.4344598Z 45d5316499db: Verifying Checksum 2022-03-17T19:08:38.4344869Z 45d5316499db: Download complete 2022-03-17T19:08:38.4947271Z cf097a682947: Download complete 2022-03-17T19:08:38.6549805Z 795311cb799f: Verifying Checksum 2022-03-17T19:08:38.7392619Z 575631837e37: Download complete 2022-03-17T19:08:38.8154423Z f415fd767169: Verifying Checksum 2022-03-17T19:08:38.8154739Z f415fd767169: Download complete 2022-03-17T19:08:39.3361575Z d9832f16d0b0: Verifying Checksum 2022-03-17T19:08:39.3361917Z d9832f16d0b0: Download complete 2022-03-17T19:08:39.5252111Z 2a708e46e4ce: Pull complete 2022-03-17T19:08:39.6436445Z a88643889c30: Pull complete 2022-03-17T19:08:45.4878518Z 8d733210f9f6: Verifying Checksum 2022-03-17T19:08:45.4878833Z 8d733210f9f6: Download complete 2022-03-17T19:08:54.3792872Z 0a97b47d3470: Pull complete 2022-03-17T19:08:54.4961944Z 95bff152db9a: Pull complete 2022-03-17T19:09:23.2146764Z f415fd767169: Pull complete 2022-03-17T19:09:25.3198277Z 2350347ab300: Pull complete 2022-03-17T19:09:32.5267934Z 9ab0eac18dff: Pull complete 2022-03-17T19:09:34.4336323Z 868da44a8860: Pull complete 2022-03-17T19:09:36.3637169Z cc0029abc03f: Pull complete 2022-03-17T19:09:38.2115814Z c45e6042be05: Pull complete 2022-03-17T19:09:41.8615485Z d1e7e8aafebd: Pull complete 2022-03-17T19:09:43.7715671Z bff23355a96d: Pull complete 2022-03-17T19:10:09.5457349Z 8d733210f9f6: Pull complete 2022-03-17T19:10:12.1539250Z e29500059cfe: Pull complete 2022-03-17T19:10:15.3289411Z a7162e106409: Pull complete 2022-03-17T19:10:17.4009974Z 156ef8f3a13c: Pull complete 2022-03-17T19:10:19.2795330Z 08ba1108f7b7: Pull complete 2022-03-17T19:10:21.1547838Z b1ca60ec900e: Pull complete 2022-03-17T19:10:25.2629122Z 9422c548b247: Pull complete 2022-03-17T19:10:27.1293199Z 2629d28d8057: Pull complete 2022-03-17T19:10:29.8228781Z 51b3d0db68b0: Pull complete 2022-03-17T19:10:32.8745498Z df925f176b66: Pull complete 2022-03-17T19:10:35.4759320Z 94e62fb0708b: Pull complete 2022-03-17T19:10:37.8278150Z 2783d99ced8a: Pull complete 2022-03-17T19:10:42.9738675Z d7107983cd6c: Pull complete 2022-03-17T19:10:45.2760692Z 9b4350e11a74: Pull complete 2022-03-17T19:10:47.0029352Z 40231bcc3933: Pull complete 2022-03-17T19:10:47.3880762Z 12611fd1614b: Pull complete 2022-03-17T19:10:47.5073387Z a8b79af452cb: Pull complete 2022-03-17T19:10:48.7258521Z 17f77bcc71e7: Pull complete 2022-03-17T19:10:48.8547823Z 88ad1b55251d: Pull complete 2022-03-17T19:10:48.9624875Z b8e74acbf1bb: Pull complete 2022-03-17T19:10:53.9956754Z 44810df15216: Pull complete 2022-03-17T19:10:54.1111538Z 4af02869d275: Pull complete 2022-03-17T19:10:54.2111738Z a23a0a601a32: Pull complete 2022-03-17T19:10:54.3424576Z 45d5316499db: Pull complete 2022-03-17T19:10:54.4757835Z cf097a682947: Pull complete 2022-03-17T19:10:55.1946389Z 795311cb799f: Pull complete 2022-03-17T19:10:55.3070326Z 575631837e37: Pull complete 2022-03-17T19:10:58.1531920Z d9832f16d0b0: Pull complete 2022-03-17T19:10:58.5167876Z Digest: sha256:e95af236903b074012c7b9a7d7543a1a50421cedce14158d8cbd1649b571f253 2022-03-17T19:10:58.5211347Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:10:58.5244025Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:10:58.5307031Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a 2022-03-17T19:10:58.5307359Z with: 2022-03-17T19:10:58.5307599Z timeout_minutes: 10 2022-03-17T19:10:58.5307830Z max_attempts: 3 2022-03-17T19:10:58.5308212Z command: set -ex bash .github/scripts/install_nvidia_utils_linux.sh echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}" 2022-03-17T19:10:58.5308596Z retry_wait_seconds: 10 2022-03-17T19:10:58.5308849Z polling_interval_seconds: 1 2022-03-17T19:10:58.5309133Z warning_on_retry: true 2022-03-17T19:10:58.5309375Z continue_on_error: false 2022-03-17T19:10:58.5309615Z env: 2022-03-17T19:10:58.5310110Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:10:58.5310615Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:10:58.5311108Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:10:58.5311557Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:10:58.5311903Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:10:58.5312151Z IN_CI: 1 2022-03-17T19:10:58.5312355Z IS_GHA: 1 2022-03-17T19:10:58.5312586Z IN_WHEEL_TEST: 1 2022-03-17T19:10:58.5312894Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:10:58.5313265Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:10:58.5313592Z PR_LABELS: [] 2022-03-17T19:10:58.5314001Z GITHUB_TOKEN: *** 2022-03-17T19:10:58.5314272Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:10:58.5314533Z PR_NUMBER: 2022-03-17T19:10:58.5314791Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:10:58.5315082Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:10:58.5315340Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:10:58.5315849Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:10:58.5316406Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:10:58.5316711Z TEST_CONFIG: distributed 2022-03-17T19:10:58.5316964Z SHARD_NUMBER: 1 2022-03-17T19:10:58.5317188Z NUM_TEST_SHARDS: 1 2022-03-17T19:10:58.5317427Z PR_BODY: 2022-03-17T19:10:58.5317639Z ##[endgroup] 2022-03-17T19:10:58.5763602Z 2022-03-17T19:10:58.5839714Z == Installing nvidia container toolkit for amzn2 == 2022-03-17T19:10:58.5842612Z + bash .github/scripts/install_nvidia_utils_linux.sh 2022-03-17T19:10:58.5843032Z + sudo yum install -y yum-utils 2022-03-17T19:10:59.0613950Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2022-03-17T19:10:59.4013101Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2022-03-17T19:10:59.4014088Z Nothing to do 2022-03-17T19:10:59.4200392Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2022-03-17T19:10:59.8691846Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2022-03-17T19:10:59.8989947Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2022-03-17T19:10:59.8990616Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2022-03-17T19:10:59.8991081Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2022-03-17T19:10:59.9131181Z + sudo yum install -y nvidia-docker2 2022-03-17T19:11:00.3580560Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2022-03-17T19:11:00.3982875Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey 2022-03-17T19:11:00.4063257Z Importing GPG key 0xF796ECB0: 2022-03-17T19:11:00.4063662Z Userid : "NVIDIA CORPORATION (Open Source Projects) " 2022-03-17T19:11:00.4064045Z Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0 2022-03-17T19:11:00.4066021Z From : https://nvidia.github.io/libnvidia-container/gpgkey 2022-03-17T19:11:00.8178216Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey 2022-03-17T19:11:00.8277670Z Importing GPG key 0xF796ECB0: 2022-03-17T19:11:00.8278085Z Userid : "NVIDIA CORPORATION (Open Source Projects) " 2022-03-17T19:11:00.8278481Z Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0 2022-03-17T19:11:00.8278977Z From : https://nvidia.github.io/nvidia-container-runtime/gpgkey 2022-03-17T19:11:01.0479695Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey 2022-03-17T19:11:01.0557496Z Importing GPG key 0xF796ECB0: 2022-03-17T19:11:01.0558225Z Userid : "NVIDIA CORPORATION (Open Source Projects) " 2022-03-17T19:11:01.0558744Z Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0 2022-03-17T19:11:01.0559222Z From : https://nvidia.github.io/nvidia-docker/gpgkey 2022-03-17T19:11:02.7246845Z Resolving Dependencies 2022-03-17T19:11:02.7253514Z --> Running transaction check 2022-03-17T19:11:02.7254052Z ---> Package nvidia-docker2.noarch 0:2.9.1-1 will be installed 2022-03-17T19:11:02.7280614Z --> Processing Dependency: nvidia-container-toolkit >= 1.8.1-1 for package: nvidia-docker2-2.9.1-1.noarch 2022-03-17T19:11:02.7606897Z --> Running transaction check 2022-03-17T19:11:02.7607853Z ---> Package nvidia-container-toolkit.x86_64 0:1.8.1-1 will be installed 2022-03-17T19:11:02.7617509Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.8.1-1.x86_64 2022-03-17T19:11:02.7733219Z --> Processing Dependency: libnvidia-container-tools >= 1.8.1-1 for package: nvidia-container-toolkit-1.8.1-1.x86_64 2022-03-17T19:11:02.7733699Z --> Running transaction check 2022-03-17T19:11:02.7735020Z ---> Package libnvidia-container-tools.x86_64 0:1.8.1-1 will be installed 2022-03-17T19:11:02.7762706Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.8.1-1 for package: libnvidia-container-tools-1.8.1-1.x86_64 2022-03-17T19:11:02.7789407Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.8.1-1.x86_64 2022-03-17T19:11:02.7790133Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.8.1-1.x86_64 2022-03-17T19:11:02.7790863Z --> Running transaction check 2022-03-17T19:11:02.7791440Z ---> Package libnvidia-container1.x86_64 0:1.8.1-1 will be installed 2022-03-17T19:11:03.0135559Z --> Finished Dependency Resolution 2022-03-17T19:11:03.0775195Z 2022-03-17T19:11:03.0775484Z Dependencies Resolved 2022-03-17T19:11:03.0787136Z 2022-03-17T19:11:03.0787412Z ================================================================================ 2022-03-17T19:11:03.0787786Z Package Arch Version Repository Size 2022-03-17T19:11:03.0788102Z ================================================================================ 2022-03-17T19:11:03.0788361Z Installing: 2022-03-17T19:11:03.0788853Z nvidia-docker2 noarch 2.9.1-1 libnvidia-container 8.7 k 2022-03-17T19:11:03.0789211Z Installing for dependencies: 2022-03-17T19:11:03.0789662Z libnvidia-container-tools x86_64 1.8.1-1 libnvidia-container 47 k 2022-03-17T19:11:03.0792401Z libnvidia-container1 x86_64 1.8.1-1 libnvidia-container 1.0 M 2022-03-17T19:11:03.0792978Z nvidia-container-toolkit x86_64 1.8.1-1 libnvidia-container 1.5 M 2022-03-17T19:11:03.0793250Z 2022-03-17T19:11:03.0793375Z Transaction Summary 2022-03-17T19:11:03.0793819Z ================================================================================ 2022-03-17T19:11:03.0794661Z Install 1 Package (+3 Dependent packages) 2022-03-17T19:11:03.0794875Z 2022-03-17T19:11:03.0795112Z Total download size: 2.5 M 2022-03-17T19:11:03.0795609Z Installed size: 7.4 M 2022-03-17T19:11:03.0796227Z Downloading packages: 2022-03-17T19:11:03.1789126Z -------------------------------------------------------------------------------- 2022-03-17T19:11:03.1789749Z Total 26 MB/s | 2.5 MB 00:00 2022-03-17T19:11:03.1833822Z Running transaction check 2022-03-17T19:11:03.1895460Z Running transaction test 2022-03-17T19:11:03.2047979Z Transaction test succeeded 2022-03-17T19:11:03.2051474Z Running transaction 2022-03-17T19:11:03.4296017Z Installing : libnvidia-container1-1.8.1-1.x86_64 1/4 2022-03-17T19:11:03.6266723Z Installing : libnvidia-container-tools-1.8.1-1.x86_64 2/4 2022-03-17T19:11:03.6457716Z Installing : nvidia-container-toolkit-1.8.1-1.x86_64 3/4 2022-03-17T19:11:03.6786943Z Installing : nvidia-docker2-2.9.1-1.noarch 4/4 2022-03-17T19:11:03.6884409Z Verifying : nvidia-container-toolkit-1.8.1-1.x86_64 1/4 2022-03-17T19:11:03.6975286Z Verifying : libnvidia-container1-1.8.1-1.x86_64 2/4 2022-03-17T19:11:03.7065450Z Verifying : libnvidia-container-tools-1.8.1-1.x86_64 3/4 2022-03-17T19:11:03.7678632Z Verifying : nvidia-docker2-2.9.1-1.noarch 4/4 2022-03-17T19:11:03.7678938Z 2022-03-17T19:11:03.7679049Z Installed: 2022-03-17T19:11:03.7679413Z nvidia-docker2.noarch 0:2.9.1-1 2022-03-17T19:11:03.7679640Z 2022-03-17T19:11:03.7679766Z Dependency Installed: 2022-03-17T19:11:03.7680171Z libnvidia-container-tools.x86_64 0:1.8.1-1 2022-03-17T19:11:03.7680621Z libnvidia-container1.x86_64 0:1.8.1-1 2022-03-17T19:11:03.7681081Z nvidia-container-toolkit.x86_64 0:1.8.1-1 2022-03-17T19:11:03.7681290Z 2022-03-17T19:11:03.7681397Z Complete! 2022-03-17T19:11:03.8584894Z + sudo systemctl restart docker 2022-03-17T19:11:14.3234944Z == Installing nvidia driver NVIDIA-Linux-x86_64-495.44.run == 2022-03-17T19:11:14.3236555Z + sudo yum groupinstall -y 'Development Tools' 2022-03-17T19:11:14.7890798Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2022-03-17T19:11:15.7860342Z Resolving Dependencies 2022-03-17T19:11:15.7866470Z --> Running transaction check 2022-03-17T19:11:15.7869475Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed 2022-03-17T19:11:15.8102681Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch 2022-03-17T19:11:15.8427439Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch 2022-03-17T19:11:15.8434101Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed 2022-03-17T19:11:15.8497958Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch 2022-03-17T19:11:15.8506128Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch 2022-03-17T19:11:15.8518046Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed 2022-03-17T19:11:15.8626875Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed 2022-03-17T19:11:15.8636719Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed 2022-03-17T19:11:15.8689061Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed 2022-03-17T19:11:15.8701492Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed 2022-03-17T19:11:15.8712770Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed 2022-03-17T19:11:15.8807957Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed 2022-03-17T19:11:15.8973182Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed 2022-03-17T19:11:15.9005165Z ---> Package gcc.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:15.9163389Z --> Processing Dependency: cpp = 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9179763Z --> Processing Dependency: libsanitizer >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9227398Z --> Processing Dependency: libquadmath >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9272418Z --> Processing Dependency: libmpx >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9321448Z --> Processing Dependency: libitm >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9367053Z --> Processing Dependency: libcilkrts >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9414248Z --> Processing Dependency: libatomic >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9462569Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9592201Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9616177Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9640139Z ---> Package gcc-c++.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:15.9676717Z ---> Package gcc-gfortran.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:15.9721340Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-13.amzn2.x86_64 2022-03-17T19:11:15.9776272Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed 2022-03-17T19:11:15.9800411Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed 2022-03-17T19:11:15.9854950Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch 2022-03-17T19:11:15.9871753Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch 2022-03-17T19:11:15.9893370Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed 2022-03-17T19:11:15.9932380Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed 2022-03-17T19:11:15.9974167Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed 2022-03-17T19:11:16.0006195Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed 2022-03-17T19:11:16.0055987Z ---> Package rpm-build.x86_64 0:4.11.3-40.amzn2.0.6 will be installed 2022-03-17T19:11:16.0292009Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-40.amzn2.0.6.x86_64 2022-03-17T19:11:16.0313039Z ---> Package rpm-sign.x86_64 0:4.11.3-40.amzn2.0.6 will be installed 2022-03-17T19:11:16.0348452Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed 2022-03-17T19:11:16.0513536Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0534973Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0536228Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0537226Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0537881Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0538544Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0539176Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0539792Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0540660Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0541450Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0542372Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0543426Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0544159Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0544797Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0545408Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0546015Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0567448Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0591180Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64 2022-03-17T19:11:16.0618045Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed 2022-03-17T19:11:16.0646962Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.10 will be installed 2022-03-17T19:11:16.0687485Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch 2022-03-17T19:11:16.0708376Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch 2022-03-17T19:11:16.0722918Z ---> Package systemtap.x86_64 0:4.4-1.amzn2.0.2 will be installed 2022-03-17T19:11:16.0740053Z --> Processing Dependency: systemtap-devel = 4.4-1.amzn2.0.2 for package: systemtap-4.4-1.amzn2.0.2.x86_64 2022-03-17T19:11:16.0753754Z --> Processing Dependency: systemtap-client = 4.4-1.amzn2.0.2 for package: systemtap-4.4-1.amzn2.0.2.x86_64 2022-03-17T19:11:16.0765276Z --> Running transaction check 2022-03-17T19:11:16.0768462Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed 2022-03-17T19:11:16.0851872Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed 2022-03-17T19:11:16.0899625Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64 2022-03-17T19:11:16.0916250Z ---> Package cpp.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:16.1003213Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed 2022-03-17T19:11:16.1018966Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed 2022-03-17T19:11:16.1099051Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed 2022-03-17T19:11:16.1165501Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64 2022-03-17T19:11:16.1174785Z ---> Package glibc-devel.x86_64 0:2.26-57.amzn2 will be installed 2022-03-17T19:11:16.1297139Z --> Processing Dependency: glibc-headers = 2.26-57.amzn2 for package: glibc-devel-2.26-57.amzn2.x86_64 2022-03-17T19:11:16.1322179Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-57.amzn2.x86_64 2022-03-17T19:11:16.1322885Z ---> Package libatomic.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:16.1346308Z ---> Package libcilkrts.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:16.1386948Z ---> Package libgfortran.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:16.1434021Z ---> Package libitm.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:16.1460529Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed 2022-03-17T19:11:16.1480939Z ---> Package libmpx.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:16.1506167Z ---> Package libquadmath.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:16.1544113Z ---> Package libsanitizer.x86_64 0:7.3.1-13.amzn2 will be installed 2022-03-17T19:11:16.1605801Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed 2022-03-17T19:11:16.1630739Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed 2022-03-17T19:11:16.1662421Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed 2022-03-17T19:11:16.1742556Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2022-03-17T19:11:16.1777567Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2022-03-17T19:11:16.1778767Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2022-03-17T19:11:16.1801013Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2022-03-17T19:11:16.1821042Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64 2022-03-17T19:11:16.1827981Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed 2022-03-17T19:11:16.1884380Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed 2022-03-17T19:11:16.2029009Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed 2022-03-17T19:11:16.2043854Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed 2022-03-17T19:11:16.2066739Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed 2022-03-17T19:11:16.2068001Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed 2022-03-17T19:11:16.2116063Z ---> Package systemtap-client.x86_64 0:4.4-1.amzn2.0.2 will be installed 2022-03-17T19:11:16.2365682Z --> Processing Dependency: mokutil for package: systemtap-client-4.4-1.amzn2.0.2.x86_64 2022-03-17T19:11:16.2381776Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.4-1.amzn2.0.2.x86_64 2022-03-17T19:11:16.2411800Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.4-1.amzn2.0.2.x86_64 2022-03-17T19:11:16.2412576Z ---> Package systemtap-devel.x86_64 0:4.4-1.amzn2.0.2 will be installed 2022-03-17T19:11:16.2535647Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.4-1.amzn2.0.2.x86_64 2022-03-17T19:11:16.3413312Z --> Running transaction check 2022-03-17T19:11:16.3414896Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed 2022-03-17T19:11:16.3428518Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed 2022-03-17T19:11:16.3462905Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed 2022-03-17T19:11:16.3464112Z ---> Package glibc-headers.x86_64 0:2.26-57.amzn2 will be installed 2022-03-17T19:11:16.3506827Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-57.amzn2.x86_64 2022-03-17T19:11:16.4421047Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-57.amzn2.x86_64 2022-03-17T19:11:16.4422049Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2 will be installed 2022-03-17T19:11:16.4494233Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.x86_64 2022-03-17T19:11:16.4524167Z ---> Package kernel-devel.x86_64 0:4.14.268-205.500.amzn2 will be installed 2022-03-17T19:11:16.4554882Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.268-205.500.amzn2.x86_64 2022-03-17T19:11:16.4577582Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed 2022-03-17T19:11:16.4617820Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64 2022-03-17T19:11:16.4638930Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed 2022-03-17T19:11:16.4687880Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 2022-03-17T19:11:16.4711395Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 2022-03-17T19:11:16.4712551Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 2022-03-17T19:11:16.4713359Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed 2022-03-17T19:11:16.4733686Z --> Running transaction check 2022-03-17T19:11:16.4734656Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed 2022-03-17T19:11:16.4761445Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed 2022-03-17T19:11:16.4774911Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64 2022-03-17T19:11:16.4797736Z ---> Package kernel-headers.x86_64 0:4.14.268-205.500.amzn2 will be installed 2022-03-17T19:11:16.4798865Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed 2022-03-17T19:11:16.4827355Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed 2022-03-17T19:11:16.4893459Z --> Running transaction check 2022-03-17T19:11:16.4894075Z ---> Package zlib-devel.x86_64 0:1.2.7-18.amzn2 will be installed 2022-03-17T19:11:16.7443808Z --> Finished Dependency Resolution 2022-03-17T19:11:16.8538552Z 2022-03-17T19:11:16.8538872Z Dependencies Resolved 2022-03-17T19:11:16.8652533Z 2022-03-17T19:11:16.8652729Z ================================================================================ 2022-03-17T19:11:16.8653239Z Package Arch Version Repository Size 2022-03-17T19:11:16.8653939Z ================================================================================ 2022-03-17T19:11:16.8654604Z Installing for group install "Development Tools": 2022-03-17T19:11:16.8655661Z autoconf noarch 2.69-11.amzn2 amzn2-core 701 k 2022-03-17T19:11:16.8656525Z automake noarch 1.13.4-3.1.amzn2 amzn2-core 679 k 2022-03-17T19:11:16.8656970Z bison x86_64 3.0.4-6.amzn2.0.2 amzn2-core 674 k 2022-03-17T19:11:16.8657377Z byacc x86_64 1.9.20130304-3.amzn2.0.2 amzn2-core 66 k 2022-03-17T19:11:16.8657862Z cscope x86_64 15.8-10.amzn2.0.2 amzn2-core 204 k 2022-03-17T19:11:16.8658272Z ctags x86_64 5.8-13.amzn2.0.2 amzn2-core 157 k 2022-03-17T19:11:16.8658701Z diffstat x86_64 1.57-4.amzn2.0.2 amzn2-core 35 k 2022-03-17T19:11:16.8659179Z doxygen x86_64 1:1.8.5-4.amzn2 amzn2-core 3.5 M 2022-03-17T19:11:16.8659582Z elfutils x86_64 0.176-2.amzn2 amzn2-core 307 k 2022-03-17T19:11:16.8659950Z flex x86_64 2.5.37-3.amzn2.0.3 amzn2-core 291 k 2022-03-17T19:11:16.8660759Z gcc x86_64 7.3.1-13.amzn2 amzn2-core 22 M 2022-03-17T19:11:16.8661211Z gcc-c++ x86_64 7.3.1-13.amzn2 amzn2-core 13 M 2022-03-17T19:11:16.8661629Z gcc-gfortran x86_64 7.3.1-13.amzn2 amzn2-core 11 M 2022-03-17T19:11:16.8662250Z indent x86_64 2.2.11-13.amzn2.0.2 amzn2-core 150 k 2022-03-17T19:11:16.8662998Z intltool noarch 0.50.2-7.amzn2 amzn2-core 59 k 2022-03-17T19:11:16.8663620Z libtool x86_64 2.4.2-22.2.amzn2.0.2 amzn2-core 588 k 2022-03-17T19:11:16.8664018Z patch x86_64 2.7.1-12.amzn2.0.2 amzn2-core 110 k 2022-03-17T19:11:16.8664583Z patchutils x86_64 0.3.3-4.amzn2.0.1 amzn2-core 104 k 2022-03-17T19:11:16.8665034Z rcs x86_64 5.9.0-5.amzn2.0.2 amzn2-core 231 k 2022-03-17T19:11:16.8665455Z rpm-build x86_64 4.11.3-40.amzn2.0.6 amzn2-core 149 k 2022-03-17T19:11:16.8666095Z rpm-sign x86_64 4.11.3-40.amzn2.0.6 amzn2-core 49 k 2022-03-17T19:11:16.8666462Z subversion x86_64 1.7.14-16.amzn2.0.1 amzn2-core 1.0 M 2022-03-17T19:11:16.8666885Z swig x86_64 3.0.12-11.amzn2.0.3 amzn2-core 1.4 M 2022-03-17T19:11:16.8667504Z system-rpm-config noarch 9.1.0-76.amzn2.0.10 amzn2-core 80 k 2022-03-17T19:11:16.8667959Z systemtap x86_64 4.4-1.amzn2.0.2 amzn2-core 12 k 2022-03-17T19:11:16.8668273Z Installing for dependencies: 2022-03-17T19:11:16.8668662Z apr x86_64 1.7.0-9.amzn2 amzn2-core 122 k 2022-03-17T19:11:16.8669082Z apr-util x86_64 1.6.1-5.amzn2.0.2 amzn2-core 99 k 2022-03-17T19:11:16.8669597Z apr-util-bdb x86_64 1.6.1-5.amzn2.0.2 amzn2-core 19 k 2022-03-17T19:11:16.8670056Z avahi-libs x86_64 0.6.31-20.amzn2 amzn2-core 61 k 2022-03-17T19:11:16.8670481Z cpp x86_64 7.3.1-13.amzn2 amzn2-core 9.2 M 2022-03-17T19:11:16.8670882Z dwz x86_64 0.11-3.amzn2.0.3 amzn2-core 98 k 2022-03-17T19:11:16.8671309Z efivar-libs x86_64 31-4.amzn2.0.4 amzn2-core 68 k 2022-03-17T19:11:16.8671756Z elfutils-libelf-devel x86_64 0.176-2.amzn2 amzn2-core 40 k 2022-03-17T19:11:16.8672206Z gdb x86_64 8.0.1-36.amzn2.0.1 amzn2-core 3.1 M 2022-03-17T19:11:16.8672653Z gettext-common-devel noarch 0.19.8.1-3.amzn2 amzn2-core 410 k 2022-03-17T19:11:16.8673096Z gettext-devel x86_64 0.19.8.1-3.amzn2 amzn2-core 320 k 2022-03-17T19:11:16.8673532Z glibc-devel x86_64 2.26-57.amzn2 amzn2-core 994 k 2022-03-17T19:11:16.8673952Z glibc-headers x86_64 2.26-57.amzn2 amzn2-core 514 k 2022-03-17T19:11:16.8674384Z gnutls x86_64 3.3.29-9.amzn2 amzn2-core 673 k 2022-03-17T19:11:16.8674813Z kernel-devel x86_64 4.14.268-205.500.amzn2 amzn2-core 13 M 2022-03-17T19:11:16.8675238Z kernel-headers x86_64 4.14.268-205.500.amzn2 amzn2-core 1.2 M 2022-03-17T19:11:16.8675678Z libatomic x86_64 7.3.1-13.amzn2 amzn2-core 46 k 2022-03-17T19:11:16.8676110Z libcilkrts x86_64 7.3.1-13.amzn2 amzn2-core 85 k 2022-03-17T19:11:16.8676526Z libgfortran x86_64 7.3.1-13.amzn2 amzn2-core 536 k 2022-03-17T19:11:16.8676947Z libitm x86_64 7.3.1-13.amzn2 amzn2-core 84 k 2022-03-17T19:11:16.8677347Z libmodman x86_64 2.0.1-8.amzn2.0.2 amzn2-core 29 k 2022-03-17T19:11:16.8677764Z libmpc x86_64 1.0.1-3.amzn2.0.2 amzn2-core 52 k 2022-03-17T19:11:16.8678189Z libmpx x86_64 7.3.1-13.amzn2 amzn2-core 51 k 2022-03-17T19:11:16.8678597Z libproxy x86_64 0.4.11-10.amzn2.0.3 amzn2-core 61 k 2022-03-17T19:11:16.8679029Z libquadmath x86_64 7.3.1-13.amzn2 amzn2-core 189 k 2022-03-17T19:11:16.8679445Z libsanitizer x86_64 7.3.1-13.amzn2 amzn2-core 641 k 2022-03-17T19:11:16.8679867Z m4 x86_64 1.4.16-10.amzn2.0.2 amzn2-core 256 k 2022-03-17T19:11:16.8680310Z mokutil x86_64 1:0.3.0-10.amzn2.0.1 amzn2-core 39 k 2022-03-17T19:11:16.8680712Z mpfr x86_64 3.1.1-4.amzn2.0.2 amzn2-core 208 k 2022-03-17T19:11:16.8681125Z neon x86_64 0.30.0-3.amzn2.0.2 amzn2-core 166 k 2022-03-17T19:11:16.8681532Z pakchois x86_64 0.4-10.amzn2.0.2 amzn2-core 14 k 2022-03-17T19:11:16.8681985Z perl-Data-Dumper x86_64 2.145-3.amzn2.0.2 amzn2-core 48 k 2022-03-17T19:11:16.8682444Z perl-Test-Harness noarch 3.28-3.amzn2 amzn2-core 302 k 2022-03-17T19:11:16.8682890Z perl-Thread-Queue noarch 3.02-2.amzn2 amzn2-core 17 k 2022-03-17T19:11:16.8683363Z perl-XML-Parser x86_64 2.41-10.amzn2.0.2 amzn2-core 223 k 2022-03-17T19:11:16.8683878Z perl-srpm-macros noarch 1-8.amzn2.0.1 amzn2-core 4.7 k 2022-03-17T19:11:16.8684342Z subversion-libs x86_64 1.7.14-16.amzn2.0.1 amzn2-core 912 k 2022-03-17T19:11:16.8684791Z systemtap-client x86_64 4.4-1.amzn2.0.2 amzn2-core 3.7 M 2022-03-17T19:11:16.8685222Z systemtap-devel x86_64 4.4-1.amzn2.0.2 amzn2-core 2.3 M 2022-03-17T19:11:16.8685658Z trousers x86_64 0.3.14-2.amzn2.0.2 amzn2-core 294 k 2022-03-17T19:11:16.8686121Z zlib-devel x86_64 1.2.7-18.amzn2 amzn2-core 50 k 2022-03-17T19:11:16.8686337Z 2022-03-17T19:11:16.8686453Z Transaction Summary 2022-03-17T19:11:16.8686734Z ================================================================================ 2022-03-17T19:11:16.8687027Z Install 25 Packages (+41 Dependent packages) 2022-03-17T19:11:16.8687226Z 2022-03-17T19:11:16.8687348Z Total download size: 96 M 2022-03-17T19:11:16.8687596Z Installed size: 303 M 2022-03-17T19:11:16.8687867Z Downloading packages: 2022-03-17T19:11:16.8708553Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2022-03-17T19:11:18.1014968Z -------------------------------------------------------------------------------- 2022-03-17T19:11:18.1015477Z Total 78 MB/s | 96 MB 00:01 2022-03-17T19:11:18.2080859Z Running transaction check 2022-03-17T19:11:18.2859040Z Running transaction test 2022-03-17T19:11:18.7214492Z Transaction test succeeded 2022-03-17T19:11:18.7217981Z Running transaction 2022-03-17T19:11:19.2632912Z Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64 1/66 2022-03-17T19:11:19.3227453Z Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64 2/66 2022-03-17T19:11:19.3858201Z Installing : m4-1.4.16-10.amzn2.0.2.x86_64 3/66 2022-03-17T19:11:19.4303210Z Installing : apr-1.7.0-9.amzn2.x86_64 4/66 2022-03-17T19:11:19.4592141Z Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64 5/66 2022-03-17T19:11:19.4920098Z Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64 6/66 2022-03-17T19:11:19.5342882Z Installing : avahi-libs-0.6.31-20.amzn2.x86_64 7/66 2022-03-17T19:11:19.5747694Z Installing : libquadmath-7.3.1-13.amzn2.x86_64 8/66 2022-03-17T19:11:19.5958823Z Installing : patch-2.7.1-12.amzn2.0.2.x86_64 9/66 2022-03-17T19:11:19.6784047Z Installing : perl-Thread-Queue-3.02-2.amzn2.noarch 10/66 2022-03-17T19:11:20.7549288Z Installing : libgfortran-7.3.1-13.amzn2.x86_64 11/66 2022-03-17T19:11:20.7894274Z Installing : cpp-7.3.1-13.amzn2.x86_64 12/66 2022-03-17T19:11:20.8216156Z Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64 13/66 2022-03-17T19:11:20.8808167Z Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64 14/66 2022-03-17T19:11:20.9385039Z Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64 15/66 2022-03-17T19:11:20.9671501Z Installing : elfutils-0.176-2.amzn2.x86_64 16/66 2022-03-17T19:11:21.0040305Z Installing : libcilkrts-7.3.1-13.amzn2.x86_64 17/66 2022-03-17T19:11:21.0285205Z Installing : dwz-0.11-3.amzn2.0.3.x86_64 18/66 2022-03-17T19:11:21.0584143Z Installing : efivar-libs-31-4.amzn2.0.4.x86_64 19/66 2022-03-17T19:11:21.7642676Z Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 20/66 2022-03-17T19:11:21.8817439Z Installing : systemtap-client-4.4-1.amzn2.0.2.x86_64 21/66 2022-03-17T19:11:22.0603709Z Installing : trousers-0.3.14-2.amzn2.0.2.x86_64 22/66 2022-03-17T19:11:22.0927348Z Installing : gnutls-3.3.29-9.amzn2.x86_64 23/66 2022-03-17T19:11:22.5090677Z Installing : libatomic-7.3.1-13.amzn2.x86_64 24/66 2022-03-17T19:11:22.5420286Z Installing : gdb-8.0.1-36.amzn2.0.1.x86_64 25/66 2022-03-17T19:11:22.5740558Z Installing : libmpx-7.3.1-13.amzn2.x86_64 26/66 2022-03-17T19:11:22.6026213Z Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch 27/66 2022-03-17T19:11:22.6293160Z Installing : system-rpm-config-9.1.0-76.amzn2.0.10.noarch 28/66 2022-03-17T19:11:22.7233886Z Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64 29/66 2022-03-17T19:11:22.7724970Z Installing : autoconf-2.69-11.amzn2.noarch 30/66 2022-03-17T19:11:22.8500011Z Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch 31/66 2022-03-17T19:11:22.8994670Z Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64 32/66 2022-03-17T19:11:22.9350725Z Installing : libitm-7.3.1-13.amzn2.x86_64 33/66 2022-03-17T19:11:22.9564264Z Installing : zlib-devel-1.2.7-18.amzn2.x86_64 34/66 2022-03-17T19:11:23.0233310Z Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64 35/66 2022-03-17T19:11:23.1299290Z Installing : perl-Test-Harness-3.28-3.amzn2.noarch 36/66 2022-03-17T19:11:23.1714421Z Installing : automake-1.13.4-3.1.amzn2.noarch 37/66 2022-03-17T19:11:23.2318990Z Installing : pakchois-0.4-10.amzn2.0.2.x86_64 38/66 2022-03-17T19:11:23.3911807Z Installing : neon-0.30.0-3.amzn2.0.2.x86_64 39/66 2022-03-17T19:11:23.7037927Z Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64 40/66 2022-03-17T19:11:23.8707519Z Installing : kernel-headers-4.14.268-205.500.amzn2.x86_64 41/66 2022-03-17T19:11:24.0028220Z Installing : glibc-headers-2.26-57.amzn2.x86_64 42/66 2022-03-17T19:11:24.1249866Z Installing : glibc-devel-2.26-57.amzn2.x86_64 43/66 2022-03-17T19:11:26.2176518Z Installing : libsanitizer-7.3.1-13.amzn2.x86_64 44/66 2022-03-17T19:11:30.0787277Z Installing : gcc-7.3.1-13.amzn2.x86_64 45/66 2022-03-17T19:11:40.8526399Z Installing : kernel-devel-4.14.268-205.500.amzn2.x86_64 46/66 2022-03-17T19:11:40.8900716Z Installing : systemtap-devel-4.4-1.amzn2.0.2.x86_64 47/66 2022-03-17T19:11:42.1261479Z Installing : systemtap-4.4-1.amzn2.0.2.x86_64 48/66 2022-03-17T19:11:43.7586132Z Installing : gcc-gfortran-7.3.1-13.amzn2.x86_64 49/66 2022-03-17T19:11:43.8920321Z Installing : gcc-c++-7.3.1-13.amzn2.x86_64 50/66 2022-03-17T19:11:44.0687582Z Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64 51/66 2022-03-17T19:11:44.1814580Z Installing : subversion-1.7.14-16.amzn2.0.1.x86_64 52/66 2022-03-17T19:11:44.2200171Z Installing : intltool-0.50.2-7.amzn2.noarch 53/66 2022-03-17T19:11:44.2757154Z Installing : rpm-build-4.11.3-40.amzn2.0.6.x86_64 54/66 2022-03-17T19:11:44.3815906Z Installing : flex-2.5.37-3.amzn2.0.3.x86_64 55/66 2022-03-17T19:11:44.4447741Z Installing : bison-3.0.4-6.amzn2.0.2.x86_64 56/66 2022-03-17T19:11:44.4925184Z Installing : rcs-5.9.0-5.amzn2.0.2.x86_64 57/66 2022-03-17T19:11:44.5409525Z Installing : indent-2.2.11-13.amzn2.0.2.x86_64 58/66 2022-03-17T19:11:44.5663278Z Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64 59/66 2022-03-17T19:11:45.2383427Z Installing : rpm-sign-4.11.3-40.amzn2.0.6.x86_64 60/66 2022-03-17T19:11:45.2819414Z Installing : 1:doxygen-1.8.5-4.amzn2.x86_64 61/66 2022-03-17T19:11:45.3371514Z Installing : diffstat-1.57-4.amzn2.0.2.x86_64 62/66 2022-03-17T19:11:45.3770179Z Installing : cscope-15.8-10.amzn2.0.2.x86_64 63/66 2022-03-17T19:11:45.6826100Z Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64 64/66 2022-03-17T19:11:45.7397507Z Installing : swig-3.0.12-11.amzn2.0.3.x86_64 65/66 2022-03-17T19:11:45.8026533Z Installing : ctags-5.8-13.amzn2.0.2.x86_64 66/66 2022-03-17T19:11:45.8139501Z Verifying : systemtap-4.4-1.amzn2.0.2.x86_64 1/66 2022-03-17T19:11:45.8238648Z Verifying : perl-Thread-Queue-3.02-2.amzn2.noarch 2/66 2022-03-17T19:11:45.8320987Z Verifying : gettext-devel-0.19.8.1-3.amzn2.x86_64 3/66 2022-03-17T19:11:45.8405232Z Verifying : system-rpm-config-9.1.0-76.amzn2.0.10.noarch 4/66 2022-03-17T19:11:45.8493439Z Verifying : patch-2.7.1-12.amzn2.0.2.x86_64 5/66 2022-03-17T19:11:45.8580149Z Verifying : libsanitizer-7.3.1-13.amzn2.x86_64 6/66 2022-03-17T19:11:45.8666830Z Verifying : flex-2.5.37-3.amzn2.0.3.x86_64 7/66 2022-03-17T19:11:45.8769755Z Verifying : kernel-headers-4.14.268-205.500.amzn2.x86_64 8/66 2022-03-17T19:11:45.8866122Z Verifying : libmpc-1.0.1-3.amzn2.0.2.x86_64 9/66 2022-03-17T19:11:45.8961063Z Verifying : gcc-gfortran-7.3.1-13.amzn2.x86_64 10/66 2022-03-17T19:11:45.9041770Z Verifying : ctags-5.8-13.amzn2.0.2.x86_64 11/66 2022-03-17T19:11:45.9201799Z Verifying : swig-3.0.12-11.amzn2.0.3.x86_64 12/66 2022-03-17T19:11:45.9285310Z Verifying : byacc-1.9.20130304-3.amzn2.0.2.x86_64 13/66 2022-03-17T19:11:45.9370255Z Verifying : gcc-c++-7.3.1-13.amzn2.x86_64 14/66 2022-03-17T19:11:45.9458310Z Verifying : pakchois-0.4-10.amzn2.0.2.x86_64 15/66 2022-03-17T19:11:45.9540067Z Verifying : libproxy-0.4.11-10.amzn2.0.3.x86_64 16/66 2022-03-17T19:11:45.9646712Z Verifying : cscope-15.8-10.amzn2.0.2.x86_64 17/66 2022-03-17T19:11:45.9732825Z Verifying : rpm-build-4.11.3-40.amzn2.0.6.x86_64 18/66 2022-03-17T19:11:45.9817517Z Verifying : diffstat-1.57-4.amzn2.0.2.x86_64 19/66 2022-03-17T19:11:45.9905900Z Verifying : libquadmath-7.3.1-13.amzn2.x86_64 20/66 2022-03-17T19:11:45.9991104Z Verifying : 1:doxygen-1.8.5-4.amzn2.x86_64 21/66 2022-03-17T19:11:46.0093024Z Verifying : glibc-headers-2.26-57.amzn2.x86_64 22/66 2022-03-17T19:11:46.0178816Z Verifying : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64 23/66 2022-03-17T19:11:46.0267775Z Verifying : rpm-sign-4.11.3-40.amzn2.0.6.x86_64 24/66 2022-03-17T19:11:46.0365723Z Verifying : cpp-7.3.1-13.amzn2.x86_64 25/66 2022-03-17T19:11:46.0454710Z Verifying : systemtap-client-4.4-1.amzn2.0.2.x86_64 26/66 2022-03-17T19:11:46.0536832Z Verifying : gnutls-3.3.29-9.amzn2.x86_64 27/66 2022-03-17T19:11:46.0622780Z Verifying : perl-Test-Harness-3.28-3.amzn2.noarch 28/66 2022-03-17T19:11:46.0706099Z Verifying : autoconf-2.69-11.amzn2.noarch 29/66 2022-03-17T19:11:46.0810210Z Verifying : intltool-0.50.2-7.amzn2.noarch 30/66 2022-03-17T19:11:46.0899008Z Verifying : apr-util-1.6.1-5.amzn2.0.2.x86_64 31/66 2022-03-17T19:11:46.0986594Z Verifying : zlib-devel-1.2.7-18.amzn2.x86_64 32/66 2022-03-17T19:11:46.1080189Z Verifying : libitm-7.3.1-13.amzn2.x86_64 33/66 2022-03-17T19:11:46.1171427Z Verifying : mpfr-3.1.1-4.amzn2.0.2.x86_64 34/66 2022-03-17T19:11:46.1264165Z Verifying : gettext-common-devel-0.19.8.1-3.amzn2.noarch 35/66 2022-03-17T19:11:46.1365746Z Verifying : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64 36/66 2022-03-17T19:11:46.1460778Z Verifying : libgfortran-7.3.1-13.amzn2.x86_64 37/66 2022-03-17T19:11:46.1542374Z Verifying : perl-srpm-macros-1-8.amzn2.0.1.noarch 38/66 2022-03-17T19:11:46.1647445Z Verifying : elfutils-libelf-devel-0.176-2.amzn2.x86_64 39/66 2022-03-17T19:11:46.1744092Z Verifying : subversion-libs-1.7.14-16.amzn2.0.1.x86_64 40/66 2022-03-17T19:11:46.1843261Z Verifying : automake-1.13.4-3.1.amzn2.noarch 41/66 2022-03-17T19:11:46.1940702Z Verifying : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64 42/66 2022-03-17T19:11:46.2031890Z Verifying : avahi-libs-0.6.31-20.amzn2.x86_64 43/66 2022-03-17T19:11:46.2128243Z Verifying : bison-3.0.4-6.amzn2.0.2.x86_64 44/66 2022-03-17T19:11:46.2210107Z Verifying : libmpx-7.3.1-13.amzn2.x86_64 45/66 2022-03-17T19:11:46.2327764Z Verifying : gdb-8.0.1-36.amzn2.0.1.x86_64 46/66 2022-03-17T19:11:46.2416173Z Verifying : patchutils-0.3.3-4.amzn2.0.1.x86_64 47/66 2022-03-17T19:11:46.2521246Z Verifying : libtool-2.4.2-22.2.amzn2.0.2.x86_64 48/66 2022-03-17T19:11:46.2630039Z Verifying : indent-2.2.11-13.amzn2.0.2.x86_64 49/66 2022-03-17T19:11:46.2725159Z Verifying : subversion-1.7.14-16.amzn2.0.1.x86_64 50/66 2022-03-17T19:11:46.2814719Z Verifying : apr-1.7.0-9.amzn2.x86_64 51/66 2022-03-17T19:11:46.2904093Z Verifying : libatomic-7.3.1-13.amzn2.x86_64 52/66 2022-03-17T19:11:46.2996321Z Verifying : trousers-0.3.14-2.amzn2.0.2.x86_64 53/66 2022-03-17T19:11:46.3137256Z Verifying : kernel-devel-4.14.268-205.500.amzn2.x86_64 54/66 2022-03-17T19:11:46.3229908Z Verifying : gcc-7.3.1-13.amzn2.x86_64 55/66 2022-03-17T19:11:46.3310077Z Verifying : neon-0.30.0-3.amzn2.0.2.x86_64 56/66 2022-03-17T19:11:46.3388468Z Verifying : efivar-libs-31-4.amzn2.0.4.x86_64 57/66 2022-03-17T19:11:46.3477395Z Verifying : dwz-0.11-3.amzn2.0.3.x86_64 58/66 2022-03-17T19:11:46.3564588Z Verifying : rcs-5.9.0-5.amzn2.0.2.x86_64 59/66 2022-03-17T19:11:46.3654837Z Verifying : systemtap-devel-4.4-1.amzn2.0.2.x86_64 60/66 2022-03-17T19:11:46.3735463Z Verifying : libcilkrts-7.3.1-13.amzn2.x86_64 61/66 2022-03-17T19:11:46.3818332Z Verifying : elfutils-0.176-2.amzn2.x86_64 62/66 2022-03-17T19:11:46.3905600Z Verifying : m4-1.4.16-10.amzn2.0.2.x86_64 63/66 2022-03-17T19:11:46.4087368Z Verifying : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64 64/66 2022-03-17T19:11:46.4173817Z Verifying : glibc-devel-2.26-57.amzn2.x86_64 65/66 2022-03-17T19:11:46.4942382Z Verifying : libmodman-2.0.1-8.amzn2.0.2.x86_64 66/66 2022-03-17T19:11:46.4942642Z 2022-03-17T19:11:46.4942730Z Installed: 2022-03-17T19:11:46.4943118Z autoconf.noarch 0:2.69-11.amzn2 2022-03-17T19:11:46.4943548Z automake.noarch 0:1.13.4-3.1.amzn2 2022-03-17T19:11:46.4944906Z bison.x86_64 0:3.0.4-6.amzn2.0.2 2022-03-17T19:11:46.4945320Z byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 2022-03-17T19:11:46.4945718Z cscope.x86_64 0:15.8-10.amzn2.0.2 2022-03-17T19:11:46.4953726Z ctags.x86_64 0:5.8-13.amzn2.0.2 2022-03-17T19:11:46.4954390Z diffstat.x86_64 0:1.57-4.amzn2.0.2 2022-03-17T19:11:46.4954885Z doxygen.x86_64 1:1.8.5-4.amzn2 2022-03-17T19:11:46.4955267Z elfutils.x86_64 0:0.176-2.amzn2 2022-03-17T19:11:46.4955678Z flex.x86_64 0:2.5.37-3.amzn2.0.3 2022-03-17T19:11:46.4956091Z gcc.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4956537Z gcc-c++.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4957218Z gcc-gfortran.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4957625Z indent.x86_64 0:2.2.11-13.amzn2.0.2 2022-03-17T19:11:46.4958096Z intltool.noarch 0:0.50.2-7.amzn2 2022-03-17T19:11:46.4958523Z libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 2022-03-17T19:11:46.4958928Z patch.x86_64 0:2.7.1-12.amzn2.0.2 2022-03-17T19:11:46.4959303Z patchutils.x86_64 0:0.3.3-4.amzn2.0.1 2022-03-17T19:11:46.4959669Z rcs.x86_64 0:5.9.0-5.amzn2.0.2 2022-03-17T19:11:46.4960081Z rpm-build.x86_64 0:4.11.3-40.amzn2.0.6 2022-03-17T19:11:46.4960496Z rpm-sign.x86_64 0:4.11.3-40.amzn2.0.6 2022-03-17T19:11:46.4960906Z subversion.x86_64 0:1.7.14-16.amzn2.0.1 2022-03-17T19:11:46.4961308Z swig.x86_64 0:3.0.12-11.amzn2.0.3 2022-03-17T19:11:46.4961725Z system-rpm-config.noarch 0:9.1.0-76.amzn2.0.10 2022-03-17T19:11:46.4962178Z systemtap.x86_64 0:4.4-1.amzn2.0.2 2022-03-17T19:11:46.4962376Z 2022-03-17T19:11:46.4962504Z Dependency Installed: 2022-03-17T19:11:46.4962880Z apr.x86_64 0:1.7.0-9.amzn2 2022-03-17T19:11:46.4963291Z apr-util.x86_64 0:1.6.1-5.amzn2.0.2 2022-03-17T19:11:46.4963753Z apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 2022-03-17T19:11:46.4964167Z avahi-libs.x86_64 0:0.6.31-20.amzn2 2022-03-17T19:11:46.4964583Z cpp.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4964965Z dwz.x86_64 0:0.11-3.amzn2.0.3 2022-03-17T19:11:46.4965378Z efivar-libs.x86_64 0:31-4.amzn2.0.4 2022-03-17T19:11:46.4965820Z elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 2022-03-17T19:11:46.4966236Z gdb.x86_64 0:8.0.1-36.amzn2.0.1 2022-03-17T19:11:46.4966672Z gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 2022-03-17T19:11:46.4967106Z gettext-devel.x86_64 0:0.19.8.1-3.amzn2 2022-03-17T19:11:46.4967528Z glibc-devel.x86_64 0:2.26-57.amzn2 2022-03-17T19:11:46.4967949Z glibc-headers.x86_64 0:2.26-57.amzn2 2022-03-17T19:11:46.4968449Z gnutls.x86_64 0:3.3.29-9.amzn2 2022-03-17T19:11:46.4968865Z kernel-devel.x86_64 0:4.14.268-205.500.amzn2 2022-03-17T19:11:46.4969280Z kernel-headers.x86_64 0:4.14.268-205.500.amzn2 2022-03-17T19:11:46.4969708Z libatomic.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4970116Z libcilkrts.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4970601Z libgfortran.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4970991Z libitm.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4971397Z libmodman.x86_64 0:2.0.1-8.amzn2.0.2 2022-03-17T19:11:46.4971789Z libmpc.x86_64 0:1.0.1-3.amzn2.0.2 2022-03-17T19:11:46.4972192Z libmpx.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4972586Z libproxy.x86_64 0:0.4.11-10.amzn2.0.3 2022-03-17T19:11:46.4972994Z libquadmath.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4973411Z libsanitizer.x86_64 0:7.3.1-13.amzn2 2022-03-17T19:11:46.4973796Z m4.x86_64 0:1.4.16-10.amzn2.0.2 2022-03-17T19:11:46.4974194Z mokutil.x86_64 1:0.3.0-10.amzn2.0.1 2022-03-17T19:11:46.4974582Z mpfr.x86_64 0:3.1.1-4.amzn2.0.2 2022-03-17T19:11:46.4974982Z neon.x86_64 0:0.30.0-3.amzn2.0.2 2022-03-17T19:11:46.4975391Z pakchois.x86_64 0:0.4-10.amzn2.0.2 2022-03-17T19:11:46.4975810Z perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 2022-03-17T19:11:46.4976267Z perl-Test-Harness.noarch 0:3.28-3.amzn2 2022-03-17T19:11:46.4976706Z perl-Thread-Queue.noarch 0:3.02-2.amzn2 2022-03-17T19:11:46.4977161Z perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 2022-03-17T19:11:46.4977620Z perl-srpm-macros.noarch 0:1-8.amzn2.0.1 2022-03-17T19:11:46.4978056Z subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 2022-03-17T19:11:46.4978492Z systemtap-client.x86_64 0:4.4-1.amzn2.0.2 2022-03-17T19:11:46.4978916Z systemtap-devel.x86_64 0:4.4-1.amzn2.0.2 2022-03-17T19:11:46.4979340Z trousers.x86_64 0:0.3.14-2.amzn2.0.2 2022-03-17T19:11:46.4979751Z zlib-devel.x86_64 0:1.2.7-18.amzn2 2022-03-17T19:11:46.4979934Z 2022-03-17T19:11:46.4980039Z Complete! 2022-03-17T19:11:46.5317792Z ++ uname -r 2022-03-17T19:11:46.5323111Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64' 2022-03-17T19:11:47.0385675Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2022-03-17T19:11:47.0542252Z Existing lock /var/run/yum.pid: another copy is running as pid 34641. 2022-03-17T19:11:47.0543016Z Another app is currently holding the yum lock; waiting for it to exit... 2022-03-17T19:11:47.0550646Z The other application is: yum 2022-03-17T19:11:47.0551280Z Memory : 92 M RSS (309 MB VSZ) 2022-03-17T19:11:47.0552810Z Started: Thu Mar 17 19:11:45 2022 - 00:02 ago 2022-03-17T19:11:47.0553285Z State : Running, pid: 34641 2022-03-17T19:11:49.0577394Z Another app is currently holding the yum lock; waiting for it to exit... 2022-03-17T19:11:49.0583791Z The other application is: yum 2022-03-17T19:11:49.0584607Z Memory : 152 M RSS (370 MB VSZ) 2022-03-17T19:11:49.0585603Z Started: Thu Mar 17 19:11:45 2022 - 00:04 ago 2022-03-17T19:11:49.0586266Z State : Running, pid: 34641 2022-03-17T19:11:52.3091078Z Resolving Dependencies 2022-03-17T19:11:52.3097355Z --> Running transaction check 2022-03-17T19:11:52.3098148Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed 2022-03-17T19:11:52.5977669Z --> Finished Dependency Resolution 2022-03-17T19:11:52.7126973Z 2022-03-17T19:11:52.7127460Z Dependencies Resolved 2022-03-17T19:11:52.7132735Z 2022-03-17T19:11:52.7133284Z ================================================================================ 2022-03-17T19:11:52.7133808Z Package Arch Version Repository Size 2022-03-17T19:11:52.7134140Z ================================================================================ 2022-03-17T19:11:52.7134377Z Installing: 2022-03-17T19:11:52.7134888Z kernel-devel x86_64 4.14.252-195.483.amzn2 amzn2-core 13 M 2022-03-17T19:11:52.7135134Z 2022-03-17T19:11:52.7135214Z Transaction Summary 2022-03-17T19:11:52.7135492Z ================================================================================ 2022-03-17T19:11:52.7135747Z Install 1 Package 2022-03-17T19:11:52.7135905Z 2022-03-17T19:11:52.7136030Z Total download size: 13 M 2022-03-17T19:11:52.7136293Z Installed size: 60 M 2022-03-17T19:11:52.7136535Z Downloading packages: 2022-03-17T19:11:52.7145577Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2022-03-17T19:11:53.3873513Z Running transaction check 2022-03-17T19:11:53.4061820Z Running transaction test 2022-03-17T19:11:53.8137143Z Transaction test succeeded 2022-03-17T19:11:53.8141140Z Running transaction 2022-03-17T19:12:08.9791326Z Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64 1/1 2022-03-17T19:12:09.0649621Z Verifying : kernel-devel-4.14.252-195.483.amzn2.x86_64 1/1 2022-03-17T19:12:09.0649897Z 2022-03-17T19:12:09.0649972Z Installed: 2022-03-17T19:12:09.0650355Z kernel-devel.x86_64 0:4.14.252-195.483.amzn2 2022-03-17T19:12:09.0650582Z 2022-03-17T19:12:09.0650673Z Complete! 2022-03-17T19:12:09.0978251Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-495.44.run 2022-03-17T19:12:11.9513710Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm 2022-03-17T19:12:13.0203506Z Verifying archive integrity... OK 2022-03-17T19:12:34.2240451Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 495.44...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... 2022-03-17T19:12:34.3641443Z 2022-03-17T19:12:34.3642206Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver. 2022-03-17T19:12:34.3642520Z 2022-03-17T19:12:49.1247418Z 2022-03-17T19:12:49.1259665Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system. If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver. 2022-03-17T19:12:49.1260643Z 2022-03-17T19:12:57.0496714Z + sudo rm -fv /tmp/nvidia_driver 2022-03-17T19:12:57.0949182Z removed '/tmp/nvidia_driver' 2022-03-17T19:12:57.0965159Z + nvidia-smi 2022-03-17T19:13:01.6361246Z Thu Mar 17 19:13:01 2022 2022-03-17T19:13:01.6361823Z +-----------------------------------------------------------------------------+ 2022-03-17T19:13:01.6363092Z | NVIDIA-SMI 495.44 Driver Version: 495.44 CUDA Version: 11.5 | 2022-03-17T19:13:01.6364204Z |-------------------------------+----------------------+----------------------+ 2022-03-17T19:13:01.6364711Z | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2022-03-17T19:13:01.6365198Z | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2022-03-17T19:13:01.6367199Z | | | MIG M. | 2022-03-17T19:13:01.6367946Z |===============================+======================+======================| 2022-03-17T19:13:01.6413019Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2022-03-17T19:13:01.6413564Z | N/A 30C P0 37W / 150W | 0MiB / 7618MiB | 0% Default | 2022-03-17T19:13:01.6413872Z | | | N/A | 2022-03-17T19:13:01.6414363Z +-------------------------------+----------------------+----------------------+ 2022-03-17T19:13:01.6493212Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2022-03-17T19:13:01.6493747Z | N/A 25C P0 38W / 150W | 0MiB / 7618MiB | 98% Default | 2022-03-17T19:13:01.6494120Z | | | N/A | 2022-03-17T19:13:01.6494563Z +-------------------------------+----------------------+----------------------+ 2022-03-17T19:13:01.6495226Z 2022-03-17T19:13:01.6495830Z +-----------------------------------------------------------------------------+ 2022-03-17T19:13:01.6496190Z | Processes: | 2022-03-17T19:13:01.6496473Z | GPU GI CI PID Type Process name GPU Memory | 2022-03-17T19:13:01.6496806Z | ID ID Usage | 2022-03-17T19:13:01.6497099Z |=============================================================================| 2022-03-17T19:13:01.6497406Z | No running processes found | 2022-03-17T19:13:01.6497851Z +-----------------------------------------------------------------------------+ 2022-03-17T19:13:02.1784804Z + echo 'GPU_FLAG=--gpus all' 2022-03-17T19:13:02.6920292Z Command completed after 1 attempt(s). 2022-03-17T19:13:02.6920602Z 2022-03-17T19:13:02.6971259Z ##[group]Run shm_size="1g" 2022-03-17T19:13:02.6971584Z shm_size="1g" 2022-03-17T19:13:02.6971849Z case "${BUILD_ENVIRONMENT}" in 2022-03-17T19:13:02.6972121Z  *cuda*) 2022-03-17T19:13:02.6972348Z  shm_size="2g" 2022-03-17T19:13:02.6972590Z  ;; 2022-03-17T19:13:02.6972821Z  *rocm*) 2022-03-17T19:13:02.6973045Z  shm_size="8g" 2022-03-17T19:13:02.6973291Z  ;; 2022-03-17T19:13:02.6973497Z esac 2022-03-17T19:13:02.6973775Z echo "SHM_SIZE=${shm_size}" >> "${GITHUB_ENV}" 2022-03-17T19:13:02.6987620Z shell: /usr/bin/bash -e {0} 2022-03-17T19:13:02.6987878Z env: 2022-03-17T19:13:02.6988177Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:13:02.6988685Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:13:02.6989244Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:13:02.6989658Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:13:02.6990022Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:13:02.6990250Z IN_CI: 1 2022-03-17T19:13:02.6990471Z IS_GHA: 1 2022-03-17T19:13:02.6990684Z IN_WHEEL_TEST: 1 2022-03-17T19:13:02.6990987Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:13:02.6991379Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:13:02.6991827Z PR_LABELS: [] 2022-03-17T19:13:02.6992378Z GITHUB_TOKEN: *** 2022-03-17T19:13:02.6992633Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:13:02.6992894Z PR_NUMBER: 2022-03-17T19:13:02.6993168Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:13:02.6993438Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:13:02.6993713Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:13:02.6994206Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:13:02.6994762Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:13:02.6995146Z TEST_CONFIG: distributed 2022-03-17T19:13:02.6995382Z SHARD_NUMBER: 1 2022-03-17T19:13:02.6995621Z NUM_TEST_SHARDS: 1 2022-03-17T19:13:02.6995845Z PR_BODY: 2022-03-17T19:13:02.6996088Z GPU_FLAG: --gpus all 2022-03-17T19:13:02.6996317Z ##[endgroup] 2022-03-17T19:13:02.7061819Z ##[group]Run seemethere/download-artifact-s3@0504774707cbc8603d7dca922e8026eb8bf3b47b 2022-03-17T19:13:02.7062210Z with: 2022-03-17T19:13:02.7062498Z name: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:13:02.7062788Z s3-bucket: gha-artifacts 2022-03-17T19:13:02.7063053Z region: us-east-1 2022-03-17T19:13:02.7063266Z env: 2022-03-17T19:13:02.7063561Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:13:02.7064063Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:13:02.7064537Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:13:02.7064960Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:13:02.7065305Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:13:02.7065547Z IN_CI: 1 2022-03-17T19:13:02.7065749Z IS_GHA: 1 2022-03-17T19:13:02.7065979Z IN_WHEEL_TEST: 1 2022-03-17T19:13:02.7066315Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:13:02.7066693Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:13:02.7067018Z PR_LABELS: [] 2022-03-17T19:13:02.7067395Z GITHUB_TOKEN: *** 2022-03-17T19:13:02.7067667Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:13:02.7067929Z PR_NUMBER: 2022-03-17T19:13:02.7068185Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:13:02.7068478Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:13:02.7068733Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:13:02.7069236Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:13:02.7069782Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:13:02.7070085Z TEST_CONFIG: distributed 2022-03-17T19:13:02.7070336Z SHARD_NUMBER: 1 2022-03-17T19:13:02.7070555Z NUM_TEST_SHARDS: 1 2022-03-17T19:13:02.7070793Z PR_BODY: 2022-03-17T19:13:02.7071021Z GPU_FLAG: --gpus all 2022-03-17T19:13:02.7071262Z SHM_SIZE: 2g 2022-03-17T19:13:02.7071498Z ##[endgroup] 2022-03-17T19:13:03.3245720Z Found 1 objects with prefix pytorch/pytorch/2000264336/linux-xenial-cuda11.3-py3.7-gcc7/ 2022-03-17T19:13:03.3246393Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-03-17T19:13:11.9401545Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2022-03-17T19:13:11.9402134Z 2022-03-17T19:13:11.9404001Z Artifact download has finished successfully 2022-03-17T19:13:11.9527096Z ##[group]Run unzip -o artifacts.zip 2022-03-17T19:13:11.9527434Z unzip -o artifacts.zip 2022-03-17T19:13:11.9540709Z shell: /usr/bin/bash -e {0} 2022-03-17T19:13:11.9540967Z env: 2022-03-17T19:13:11.9541259Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:13:11.9541765Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:13:11.9542243Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:13:11.9542788Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:13:11.9543151Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:13:11.9543378Z IN_CI: 1 2022-03-17T19:13:11.9543596Z IS_GHA: 1 2022-03-17T19:13:11.9543809Z IN_WHEEL_TEST: 1 2022-03-17T19:13:11.9544114Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:13:11.9544484Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:13:11.9544841Z PR_LABELS: [] 2022-03-17T19:13:11.9545275Z GITHUB_TOKEN: *** 2022-03-17T19:13:11.9545546Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:13:11.9545787Z PR_NUMBER: 2022-03-17T19:13:11.9546062Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:13:11.9546339Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:13:11.9546614Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:13:11.9547124Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:13:11.9547666Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:13:11.9547988Z TEST_CONFIG: distributed 2022-03-17T19:13:11.9548223Z SHARD_NUMBER: 1 2022-03-17T19:13:11.9548465Z NUM_TEST_SHARDS: 1 2022-03-17T19:13:11.9548687Z PR_BODY: 2022-03-17T19:13:11.9548932Z GPU_FLAG: --gpus all 2022-03-17T19:13:11.9549177Z SHM_SIZE: 2g 2022-03-17T19:13:11.9549396Z ##[endgroup] 2022-03-17T19:13:11.9616920Z Archive: artifacts.zip 2022-03-17T19:13:11.9618740Z creating: dist/ 2022-03-17T19:13:14.3649360Z inflating: dist/torch-1.12.0a0+git4b4f652-cp37-cp37m-linux_x86_64.whl 2022-03-17T19:13:14.3650156Z creating: build/custom_test_artifacts/ 2022-03-17T19:13:14.3650959Z creating: build/custom_test_artifacts/custom-op-build/ 2022-03-17T19:13:14.3651858Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2022-03-17T19:13:14.3657161Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log 2022-03-17T19:13:14.3658216Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/ 2022-03-17T19:13:14.3659289Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeSystem.cmake 2022-03-17T19:13:14.3660746Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdC/ 2022-03-17T19:13:14.3661778Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdC/tmp/ 2022-03-17T19:13:14.3663459Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdC/CMakeCCompilerId.c 2022-03-17T19:13:14.3665123Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdC/a.out 2022-03-17T19:13:14.3666220Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCXX/ 2022-03-17T19:13:14.3667319Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCXX/tmp/ 2022-03-17T19:13:14.3669447Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-03-17T19:13:14.3671553Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCXX/a.out 2022-03-17T19:13:14.3673857Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_C.bin 2022-03-17T19:13:14.3675484Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeCCompiler.cmake 2022-03-17T19:13:14.3677435Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CXX.bin 2022-03-17T19:13:14.3679125Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeCXXCompiler.cmake 2022-03-17T19:13:14.3680250Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/ 2022-03-17T19:13:14.3681400Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/ 2022-03-17T19:13:14.3731779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2022-03-17T19:13:14.3733215Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2022-03-17T19:13:14.3734684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2022-03-17T19:13:14.3736182Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2022-03-17T19:13:14.3737617Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2022-03-17T19:13:14.3739068Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2022-03-17T19:13:14.3740661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2022-03-17T19:13:14.3742111Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2022-03-17T19:13:14.3743552Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2022-03-17T19:13:14.3780044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2022-03-17T19:13:14.3818053Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2022-03-17T19:13:14.3819509Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2022-03-17T19:13:14.3821185Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2022-03-17T19:13:14.3822473Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.reg.c 2022-03-17T19:13:14.3823754Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin 2022-03-17T19:13:14.3825036Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2022-03-17T19:13:14.3826224Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.o 2022-03-17T19:13:14.3827895Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/CMakeCUDACompilerId.cu 2022-03-17T19:13:14.3896990Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CompilerIdCUDA/a.out 2022-03-17T19:13:14.3966013Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CUDA.bin 2022-03-17T19:13:14.3967287Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.10.3/CMakeCUDACompiler.cmake 2022-03-17T19:13:14.3968386Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2022-03-17T19:13:14.3969465Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.c 2022-03-17T19:13:14.3970520Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.cxx 2022-03-17T19:13:14.3972061Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.bin 2022-03-17T19:13:14.3973673Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log 2022-03-17T19:13:14.3974815Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2022-03-17T19:13:14.3975777Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2022-03-17T19:13:14.3999328Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2022-03-17T19:13:14.4000486Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2022-03-17T19:13:14.4001817Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2022-03-17T19:13:14.4003030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2022-03-17T19:13:14.4004230Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2022-03-17T19:13:14.4005410Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2022-03-17T19:13:14.4006630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2022-03-17T19:13:14.4060006Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/CXX.includecache 2022-03-17T19:13:14.4078095Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.internal 2022-03-17T19:13:14.4190286Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2022-03-17T19:13:14.4191361Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2022-03-17T19:13:14.4216962Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2022-03-17T19:13:14.4218180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2022-03-17T19:13:14.4219389Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2022-03-17T19:13:14.4220869Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2022-03-17T19:13:14.4222166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2022-03-17T19:13:14.4223365Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2022-03-17T19:13:14.4224625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2022-03-17T19:13:14.4277912Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/CXX.includecache 2022-03-17T19:13:14.4295725Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.internal 2022-03-17T19:13:14.4372394Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2022-03-17T19:13:14.4373680Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-03-17T19:13:14.4374899Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2022-03-17T19:13:14.4376021Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2022-03-17T19:13:14.4377089Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2022-03-17T19:13:14.4378383Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2022-03-17T19:13:14.4379482Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2022-03-17T19:13:14.4383245Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2022-03-17T19:13:14.4384352Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2022-03-17T19:13:14.4385134Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2022-03-17T19:13:14.4477151Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2022-03-17T19:13:14.4535051Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2022-03-17T19:13:14.4535549Z creating: build/custom_test_artifacts/jit-hook-build/ 2022-03-17T19:13:14.4535977Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2022-03-17T19:13:14.4542181Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log 2022-03-17T19:13:14.4542726Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/ 2022-03-17T19:13:14.4543431Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeSystem.cmake 2022-03-17T19:13:14.4543978Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdC/ 2022-03-17T19:13:14.4544513Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdC/tmp/ 2022-03-17T19:13:14.4547945Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdC/CMakeCCompilerId.c 2022-03-17T19:13:14.4549879Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdC/a.out 2022-03-17T19:13:14.4550981Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCXX/ 2022-03-17T19:13:14.4552110Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCXX/tmp/ 2022-03-17T19:13:14.4554501Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-03-17T19:13:14.4556500Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCXX/a.out 2022-03-17T19:13:14.4559421Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_C.bin 2022-03-17T19:13:14.4560629Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeCCompiler.cmake 2022-03-17T19:13:14.4562413Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CXX.bin 2022-03-17T19:13:14.4563928Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeCXXCompiler.cmake 2022-03-17T19:13:14.4565044Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/ 2022-03-17T19:13:14.4566166Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/ 2022-03-17T19:13:14.4617145Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2022-03-17T19:13:14.4618589Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2022-03-17T19:13:14.4619997Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2022-03-17T19:13:14.4621756Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2022-03-17T19:13:14.4623208Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2022-03-17T19:13:14.4624571Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2022-03-17T19:13:14.4625986Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2022-03-17T19:13:14.4627364Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2022-03-17T19:13:14.4628799Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2022-03-17T19:13:14.4665947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2022-03-17T19:13:14.4703999Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2022-03-17T19:13:14.4705391Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2022-03-17T19:13:14.4706761Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2022-03-17T19:13:14.4708004Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.reg.c 2022-03-17T19:13:14.4709199Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin 2022-03-17T19:13:14.4710618Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2022-03-17T19:13:14.4711701Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.o 2022-03-17T19:13:14.4713646Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/CMakeCUDACompilerId.cu 2022-03-17T19:13:14.4782023Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CompilerIdCUDA/a.out 2022-03-17T19:13:14.4850271Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CUDA.bin 2022-03-17T19:13:14.4851497Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.10.3/CMakeCUDACompiler.cmake 2022-03-17T19:13:14.4852615Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2022-03-17T19:13:14.4853653Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.c 2022-03-17T19:13:14.4854605Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.cxx 2022-03-17T19:13:14.4857056Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.bin 2022-03-17T19:13:14.4857751Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log 2022-03-17T19:13:14.4858913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2022-03-17T19:13:14.4860035Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2022-03-17T19:13:14.4886190Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2022-03-17T19:13:14.4887026Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2022-03-17T19:13:14.4887677Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2022-03-17T19:13:14.4888857Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2022-03-17T19:13:14.4889534Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2022-03-17T19:13:14.4890328Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2022-03-17T19:13:14.4891016Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2022-03-17T19:13:14.4944268Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/CXX.includecache 2022-03-17T19:13:14.4961076Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.internal 2022-03-17T19:13:14.5018873Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2022-03-17T19:13:14.5019540Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-03-17T19:13:14.5020145Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2022-03-17T19:13:14.5020881Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2022-03-17T19:13:14.5021883Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2022-03-17T19:13:14.5023345Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2022-03-17T19:13:14.5023878Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2022-03-17T19:13:14.5026677Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2022-03-17T19:13:14.5027307Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2022-03-17T19:13:14.5028193Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2022-03-17T19:13:14.5073566Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2022-03-17T19:13:14.5074267Z creating: build/custom_test_artifacts/custom-backend-build/ 2022-03-17T19:13:14.5074719Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2022-03-17T19:13:14.5079640Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log 2022-03-17T19:13:14.5080224Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/ 2022-03-17T19:13:14.5080764Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeSystem.cmake 2022-03-17T19:13:14.5081329Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdC/ 2022-03-17T19:13:14.5081904Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdC/tmp/ 2022-03-17T19:13:14.5083381Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdC/CMakeCCompilerId.c 2022-03-17T19:13:14.5085072Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdC/a.out 2022-03-17T19:13:14.5085671Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCXX/ 2022-03-17T19:13:14.5086221Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCXX/tmp/ 2022-03-17T19:13:14.5087906Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCXX/CMakeCXXCompilerId.cpp 2022-03-17T19:13:14.5089189Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCXX/a.out 2022-03-17T19:13:14.5090836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_C.bin 2022-03-17T19:13:14.5091644Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeCCompiler.cmake 2022-03-17T19:13:14.5093176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CXX.bin 2022-03-17T19:13:14.5094267Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeCXXCompiler.cmake 2022-03-17T19:13:14.5094859Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/ 2022-03-17T19:13:14.5095459Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/ 2022-03-17T19:13:14.5145427Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2022-03-17T19:13:14.5146163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2022-03-17T19:13:14.5146882Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2022-03-17T19:13:14.5147657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2022-03-17T19:13:14.5148359Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2022-03-17T19:13:14.5149111Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2022-03-17T19:13:14.5149773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2022-03-17T19:13:14.5150606Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2022-03-17T19:13:14.5151624Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2022-03-17T19:13:14.5189020Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2022-03-17T19:13:14.5225910Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2022-03-17T19:13:14.5226995Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2022-03-17T19:13:14.5227727Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2022-03-17T19:13:14.5228584Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.reg.c 2022-03-17T19:13:14.5229482Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin 2022-03-17T19:13:14.5230285Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2022-03-17T19:13:14.5231211Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/tmp/a_dlink.o 2022-03-17T19:13:14.5232683Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/CMakeCUDACompilerId.cu 2022-03-17T19:13:14.5300837Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CompilerIdCUDA/a.out 2022-03-17T19:13:14.5368551Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeDetermineCompilerABI_CUDA.bin 2022-03-17T19:13:14.5369245Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.10.3/CMakeCUDACompiler.cmake 2022-03-17T19:13:14.5369803Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2022-03-17T19:13:14.5370505Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.c 2022-03-17T19:13:14.5371568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.cxx 2022-03-17T19:13:14.5373742Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.bin 2022-03-17T19:13:14.5374261Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log 2022-03-17T19:13:14.5374866Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2022-03-17T19:13:14.5375411Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2022-03-17T19:13:14.5402216Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2022-03-17T19:13:14.5402861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2022-03-17T19:13:14.5403466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2022-03-17T19:13:14.5404354Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2022-03-17T19:13:14.5405156Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2022-03-17T19:13:14.5405946Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2022-03-17T19:13:14.5406568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2022-03-17T19:13:14.5459456Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/CXX.includecache 2022-03-17T19:13:14.5476631Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.internal 2022-03-17T19:13:14.5527967Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2022-03-17T19:13:14.5528564Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2022-03-17T19:13:14.5533421Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2022-03-17T19:13:14.5534142Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2022-03-17T19:13:14.5534734Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2022-03-17T19:13:14.5535604Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2022-03-17T19:13:14.5536412Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2022-03-17T19:13:14.5537142Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2022-03-17T19:13:14.5537782Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2022-03-17T19:13:14.5545443Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/CXX.includecache 2022-03-17T19:13:14.5548989Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.internal 2022-03-17T19:13:14.5689684Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2022-03-17T19:13:14.5690367Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2022-03-17T19:13:14.5690974Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2022-03-17T19:13:14.5691524Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2022-03-17T19:13:14.5692649Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2022-03-17T19:13:14.5693985Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2022-03-17T19:13:14.5694572Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2022-03-17T19:13:14.5697509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2022-03-17T19:13:14.5698520Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2022-03-17T19:13:14.5699288Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2022-03-17T19:13:14.5814664Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2022-03-17T19:13:14.5855755Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2022-03-17T19:13:14.5856086Z creating: build/lib/ 2022-03-17T19:13:14.5856686Z inflating: build/lib/libclog.a 2022-03-17T19:13:14.5921677Z inflating: build/lib/libgtest.a 2022-03-17T19:13:14.5931700Z inflating: build/lib/libpthreadpool.a 2022-03-17T19:13:14.6019377Z inflating: build/lib/libbenchmark.a 2022-03-17T19:13:14.6125758Z inflating: build/lib/libprotobuf-lite.a 2022-03-17T19:13:14.6181548Z inflating: build/lib/libasmjit.a 2022-03-17T19:13:14.6213308Z inflating: build/lib/libtensorpipe_uv.a 2022-03-17T19:13:14.6345700Z inflating: build/lib/libgloo.a 2022-03-17T19:13:14.6365131Z inflating: build/lib/libfmt.a 2022-03-17T19:13:14.6897417Z inflating: build/lib/libprotobuf.a 2022-03-17T19:13:14.6903249Z inflating: build/lib/libbreakpad_common.a 2022-03-17T19:13:14.6903945Z inflating: build/lib/libfoxi_loader.a 2022-03-17T19:13:14.6965249Z inflating: build/lib/libc10.so 2022-03-17T19:13:14.6966246Z inflating: build/lib/libtorch_global_deps.so 2022-03-17T19:13:14.6968778Z inflating: build/lib/libcaffe2_nvrtc.so 2022-03-17T19:13:14.6978235Z inflating: build/lib/libcpuinfo.a 2022-03-17T19:13:14.6980739Z inflating: build/lib/libnnpack_reference_layers.a 2022-03-17T19:13:14.6999362Z inflating: build/lib/libgmock.a 2022-03-17T19:13:14.6999928Z inflating: build/lib/libgtest_main.a 2022-03-17T19:13:14.7009089Z inflating: build/lib/libcpuinfo_internals.a 2022-03-17T19:13:14.7009455Z inflating: build/lib/libbenchmark_main.a 2022-03-17T19:13:15.5187753Z inflating: build/lib/libdnnl.a 2022-03-17T19:13:15.5756248Z inflating: build/lib/libprotoc.a 2022-03-17T19:13:15.5772974Z inflating: build/lib/libbreakpad.a 2022-03-17T19:13:15.6425694Z inflating: build/lib/libtensorpipe.a 2022-03-17T19:13:15.6469057Z inflating: build/lib/libc10_cuda.so 2022-03-17T19:13:15.6484545Z inflating: build/lib/libqnnpack.a 2022-03-17T19:13:15.6485077Z inflating: build/lib/libgmock_main.a 2022-03-17T19:13:15.7862307Z inflating: build/lib/libfbgemm.a 2022-03-17T19:13:15.8265671Z inflating: build/lib/libkineto.a 2022-03-17T19:13:15.8289382Z inflating: build/lib/libpytorch_qnnpack.a 2022-03-17T19:13:15.8578184Z inflating: build/lib/libtensorpipe_cuda.a 2022-03-17T19:13:15.8623648Z inflating: build/lib/libcaffe2_protos.a 2022-03-17T19:13:15.8671453Z inflating: build/lib/libonnx_proto.a 2022-03-17T19:13:15.9332656Z inflating: build/lib/libonnx.a 2022-03-17T19:13:15.9355029Z inflating: build/lib/libnnpack.a 2022-03-17T19:13:15.9766781Z inflating: build/lib/libgloo_cuda.a 2022-03-17T19:13:15.9781697Z inflating: build/lib/libtest_deploy_lib.so 2022-03-17T19:13:16.4563167Z inflating: build/lib/libtorch_python_static.a 2022-03-17T19:13:17.1089165Z inflating: build/lib/libtorch_deployinterpreter.so 2022-03-17T19:13:19.0591490Z inflating: build/lib/libtorch_cpu.so 2022-03-17T19:13:19.4945192Z inflating: build/lib/libtorch_cuda_cpp.so 2022-03-17T19:13:21.0783285Z inflating: build/lib/libtorch_cuda_cu.so 2022-03-17T19:13:21.0784066Z inflating: build/lib/libtorch_cuda.so 2022-03-17T19:13:21.0785673Z inflating: build/lib/libtorch.so 2022-03-17T19:13:21.0789167Z inflating: build/lib/libc10d_cuda_test.so 2022-03-17T19:13:22.0646892Z inflating: build/lib/libtorch_cuda_linalg.so 2022-03-17T19:13:22.0671102Z inflating: build/lib/libjitbackend_test.so 2022-03-17T19:13:22.0727560Z inflating: build/lib/libtorchbind_test.so 2022-03-17T19:13:22.0758289Z inflating: build/lib/libbackend_with_compiler.so 2022-03-17T19:13:22.0763063Z inflating: build/lib/libshm.so 2022-03-17T19:13:22.7345857Z inflating: build/lib/libtorch_deploy_internal.a 2022-03-17T19:13:22.8845406Z inflating: build/lib/libtorch_python.so 2022-03-17T19:13:22.8883611Z inflating: build/lib/libnnapi_backend.so 2022-03-17T19:13:22.8883875Z creating: build/bin/ 2022-03-17T19:13:22.8895758Z inflating: build/bin/remove_dt_needed 2022-03-17T19:13:22.8952453Z inflating: build/bin/c10_registry_test 2022-03-17T19:13:22.9029529Z inflating: build/bin/c10_optional_test 2022-03-17T19:13:22.9200961Z inflating: build/bin/c10_intrusive_ptr_test 2022-03-17T19:13:22.9252396Z inflating: build/bin/c10_flags_test 2022-03-17T19:13:22.9306167Z inflating: build/bin/c10_exception_test 2022-03-17T19:13:22.9365439Z inflating: build/bin/c10_logging_test 2022-03-17T19:13:22.9422879Z inflating: build/bin/c10_complex_test 2022-03-17T19:13:22.9535527Z inflating: build/bin/c10_either_test 2022-03-17T19:13:22.9588195Z inflating: build/bin/c10_irange_test 2022-03-17T19:13:22.9645329Z inflating: build/bin/c10_bfloat16_test 2022-03-17T19:13:22.9699625Z inflating: build/bin/c10_Bitset_test 2022-03-17T19:13:22.9754994Z inflating: build/bin/c10_complex_math_test 2022-03-17T19:13:22.9812637Z inflating: build/bin/c10_InlineStreamGuard_test 2022-03-17T19:13:22.9960628Z inflating: build/bin/c10_SmallVectorTest 2022-03-17T19:13:23.0017466Z inflating: build/bin/c10_InlineDeviceGuard_test 2022-03-17T19:13:23.0075742Z inflating: build/bin/c10_typeid_test 2022-03-17T19:13:23.0134097Z inflating: build/bin/c10_SizesAndStrides_test 2022-03-17T19:13:23.0187100Z inflating: build/bin/c10_accumulate_test 2022-03-17T19:13:23.0247667Z inflating: build/bin/c10_string_view_test 2022-03-17T19:13:23.0300257Z inflating: build/bin/c10_tempfile_test 2022-03-17T19:13:23.0350889Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2022-03-17T19:13:23.0399806Z inflating: build/bin/c10_StreamGuard_test 2022-03-17T19:13:23.0463349Z inflating: build/bin/c10_ordered_preserving_dict_test 2022-03-17T19:13:23.0521865Z inflating: build/bin/c10_ThreadLocal_test 2022-03-17T19:13:23.0578763Z inflating: build/bin/c10_DispatchKeySet_test 2022-03-17T19:13:23.0632142Z inflating: build/bin/c10_DeviceGuard_test 2022-03-17T19:13:23.0684453Z inflating: build/bin/c10_C++17_test 2022-03-17T19:13:23.0733596Z inflating: build/bin/c10_TypeTraits_test 2022-03-17T19:13:23.0785575Z inflating: build/bin/c10_Device_test 2022-03-17T19:13:23.0836592Z inflating: build/bin/c10_Half_test 2022-03-17T19:13:23.0894255Z inflating: build/bin/c10_LeftRight_test 2022-03-17T19:13:23.0943700Z inflating: build/bin/c10_ConstexprCrc_test 2022-03-17T19:13:23.1007748Z inflating: build/bin/c10_Metaprogramming_test 2022-03-17T19:13:23.1056859Z inflating: build/bin/c10_Array_test 2022-03-17T19:13:23.1111094Z inflating: build/bin/c10_TypeIndex_test 2022-03-17T19:13:23.1162425Z inflating: build/bin/c10_Synchronized_test 2022-03-17T19:13:23.1214026Z inflating: build/bin/c10_TypeList_test 2022-03-17T19:13:23.1271775Z inflating: build/bin/c10_intrusive_ptr_benchmark 2022-03-17T19:13:23.1787572Z inflating: build/bin/protoc-3.13.0.0 2022-03-17T19:13:23.2304384Z inflating: build/bin/protoc 2022-03-17T19:13:23.2354138Z inflating: build/bin/c10_cuda_CUDATest 2022-03-17T19:13:23.2662571Z inflating: build/bin/vec_test_all_types_DEFAULT 2022-03-17T19:13:23.3006441Z inflating: build/bin/vec_test_all_types_AVX2 2022-03-17T19:13:23.3068615Z inflating: build/bin/TCPStoreTest 2022-03-17T19:13:23.3123362Z inflating: build/bin/FileStoreTest 2022-03-17T19:13:23.3178391Z inflating: build/bin/HashStoreTest 2022-03-17T19:13:23.3193286Z inflating: build/bin/ProcessGroupMPITest 2022-03-17T19:13:23.3196279Z inflating: build/bin/example_allreduce 2022-03-17T19:13:23.3266224Z inflating: build/bin/cuda_cub_test 2022-03-17T19:13:23.3316475Z inflating: build/bin/cuda_cudnn_test 2022-03-17T19:13:23.3370397Z inflating: build/bin/cuda_reportMemoryUsage_test 2022-03-17T19:13:23.3424574Z inflating: build/bin/cuda_vectorized_test 2022-03-17T19:13:23.3487295Z inflating: build/bin/cuda_stream_test 2022-03-17T19:13:23.3537458Z inflating: build/bin/cuda_optional_test 2022-03-17T19:13:23.3591171Z inflating: build/bin/cuda_apply_test 2022-03-17T19:13:23.3661692Z inflating: build/bin/cuda_complex_math_test 2022-03-17T19:13:23.3731948Z inflating: build/bin/cuda_distributions_test 2022-03-17T19:13:23.3784970Z inflating: build/bin/memory_overlapping_test 2022-03-17T19:13:23.3846141Z inflating: build/bin/apply_utils_test 2022-03-17T19:13:23.4142668Z inflating: build/bin/op_registration_test 2022-03-17T19:13:23.4203032Z inflating: build/bin/cuda_generator_test 2022-03-17T19:13:23.4333104Z inflating: build/bin/kernel_lambda_legacy_test 2022-03-17T19:13:23.4394482Z inflating: build/bin/type_test 2022-03-17T19:13:23.4455056Z inflating: build/bin/cpu_generator_test 2022-03-17T19:13:23.4508169Z inflating: build/bin/cuda_integer_divider_test 2022-03-17T19:13:23.4561895Z inflating: build/bin/Dimname_test 2022-03-17T19:13:23.4612245Z inflating: build/bin/dispatch_key_set_test 2022-03-17T19:13:23.4613808Z inflating: build/bin/verify_api_visibility 2022-03-17T19:13:23.4668421Z inflating: build/bin/cpu_profiling_allocator_test 2022-03-17T19:13:23.4730356Z inflating: build/bin/basic 2022-03-17T19:13:23.4788312Z inflating: build/bin/test_parallel 2022-03-17T19:13:23.4840548Z inflating: build/bin/reportMemoryUsage_test 2022-03-17T19:13:23.4908858Z inflating: build/bin/pow_test 2022-03-17T19:13:23.4959669Z inflating: build/bin/reduce_ops_test 2022-03-17T19:13:23.5008892Z inflating: build/bin/op_allowlist_test 2022-03-17T19:13:23.5110286Z inflating: build/bin/kernel_function_test 2022-03-17T19:13:23.5209444Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2022-03-17T19:13:23.5269054Z inflating: build/bin/scalar_test 2022-03-17T19:13:23.5344202Z inflating: build/bin/Dict_test 2022-03-17T19:13:23.5394874Z inflating: build/bin/variant_test 2022-03-17T19:13:23.5446062Z inflating: build/bin/cuda_half_test 2022-03-17T19:13:23.5499920Z inflating: build/bin/cuda_packedtensoraccessor_test 2022-03-17T19:13:23.5559893Z inflating: build/bin/NamedTensor_test 2022-03-17T19:13:23.5612142Z inflating: build/bin/weakref_test 2022-03-17T19:13:23.5670115Z inflating: build/bin/extension_backend_test 2022-03-17T19:13:23.5721017Z inflating: build/bin/dlconvertor_test 2022-03-17T19:13:23.5778516Z inflating: build/bin/half_test 2022-03-17T19:13:23.5830948Z inflating: build/bin/wrapdim_test 2022-03-17T19:13:23.5885157Z inflating: build/bin/inline_container_test 2022-03-17T19:13:23.5997642Z inflating: build/bin/List_test 2022-03-17T19:13:23.6055958Z inflating: build/bin/scalar_tensor_test 2022-03-17T19:13:23.6113938Z inflating: build/bin/native_test 2022-03-17T19:13:23.6193557Z inflating: build/bin/tensor_iterator_test 2022-03-17T19:13:23.6247069Z inflating: build/bin/undefined_tensor_test 2022-03-17T19:13:23.6249986Z inflating: build/bin/thread_init_test 2022-03-17T19:13:23.6347971Z inflating: build/bin/ivalue_test 2022-03-17T19:13:23.6399907Z inflating: build/bin/CppSignature_test 2022-03-17T19:13:23.6458679Z inflating: build/bin/quantized_test 2022-03-17T19:13:23.6509358Z inflating: build/bin/lazy_tensor_test 2022-03-17T19:13:23.6563624Z inflating: build/bin/math_kernel_test 2022-03-17T19:13:23.6671808Z inflating: build/bin/kernel_lambda_test 2022-03-17T19:13:23.6725803Z inflating: build/bin/memory_format_test 2022-03-17T19:13:23.6796565Z inflating: build/bin/vmap_test 2022-03-17T19:13:23.6848011Z inflating: build/bin/operators_test 2022-03-17T19:13:23.6899517Z inflating: build/bin/cuda_dlconvertor_test 2022-03-17T19:13:23.6948926Z inflating: build/bin/mobile_memory_cleanup 2022-03-17T19:13:23.7039861Z inflating: build/bin/cpu_rng_test 2022-03-17T19:13:23.7091897Z inflating: build/bin/stride_properties_test 2022-03-17T19:13:23.7157575Z inflating: build/bin/KernelFunction_test 2022-03-17T19:13:23.7218723Z inflating: build/bin/atest 2022-03-17T19:13:23.7342080Z inflating: build/bin/kernel_function_legacy_test 2022-03-17T19:13:23.7403275Z inflating: build/bin/kernel_stackbased_test 2022-03-17T19:13:23.7464031Z inflating: build/bin/cuda_atomic_ops_test 2022-03-17T19:13:23.7521954Z inflating: build/bin/backend_fallback_test 2022-03-17T19:13:23.7577606Z inflating: build/bin/broadcast_test 2022-03-17T19:13:23.7638466Z inflating: build/bin/cuda_complex_test 2022-03-17T19:13:23.7692985Z inflating: build/bin/cuda_caching_host_allocator_test 2022-03-17T19:13:23.7759505Z inflating: build/bin/ProcessGroupGlooTest 2022-03-17T19:13:23.7776765Z inflating: build/bin/tutorial_tensorexpr 2022-03-17T19:13:23.7837047Z inflating: build/bin/ProcessGroupGlooAsyncTest 2022-03-17T19:13:23.7897648Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2022-03-17T19:13:23.7960744Z inflating: build/bin/ProcessGroupNCCLTest 2022-03-17T19:13:23.8015500Z inflating: build/bin/test_dist_autograd 2022-03-17T19:13:23.8018197Z inflating: build/bin/parallel_benchmark 2022-03-17T19:13:23.8029481Z inflating: build/bin/aot_model_compiler_test 2022-03-17T19:13:23.8100782Z inflating: build/bin/test_cpp_rpc 2022-03-17T19:13:23.8171945Z inflating: build/bin/test_mobile_nnc 2022-03-17T19:13:23.8177488Z inflating: build/bin/torch_shm_manager 2022-03-17T19:13:23.8268430Z inflating: build/bin/test_lazy 2022-03-17T19:13:23.9121085Z inflating: build/bin/test_tensorexpr 2022-03-17T19:13:23.9240078Z inflating: build/bin/nvfuser_bench 2022-03-17T19:13:24.5794315Z inflating: build/bin/interactive_embedded_interpreter 2022-03-17T19:13:25.2394290Z inflating: build/bin/test_deploy_gpu 2022-03-17T19:13:25.9022043Z inflating: build/bin/test_deploy 2022-03-17T19:13:26.5579004Z inflating: build/bin/deploy_benchmark 2022-03-17T19:13:26.6464636Z inflating: build/bin/test_jit 2022-03-17T19:13:27.4276043Z inflating: build/bin/test_api 2022-03-17T19:13:27.4277176Z inflating: .pytorch-test-times.json 2022-03-17T19:13:27.4306085Z ##[group]Run sudo df -H 2022-03-17T19:13:27.4306382Z sudo df -H 2022-03-17T19:13:27.4319296Z shell: /usr/bin/bash -e {0} 2022-03-17T19:13:27.4319550Z env: 2022-03-17T19:13:27.4319842Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:13:27.4320346Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:13:27.4320840Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:13:27.4321246Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:13:27.4321605Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:13:27.4321832Z IN_CI: 1 2022-03-17T19:13:27.4322049Z IS_GHA: 1 2022-03-17T19:13:27.4322271Z IN_WHEEL_TEST: 1 2022-03-17T19:13:27.4322575Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:13:27.4322960Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:13:27.4323304Z PR_LABELS: [] 2022-03-17T19:13:27.4323724Z GITHUB_TOKEN: *** 2022-03-17T19:13:27.4323975Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:13:27.4324233Z PR_NUMBER: 2022-03-17T19:13:27.4324490Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:13:27.4324781Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:13:27.4325040Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:13:27.4325548Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:13:27.4326098Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:13:27.4326405Z TEST_CONFIG: distributed 2022-03-17T19:13:27.4326656Z SHARD_NUMBER: 1 2022-03-17T19:13:27.4326884Z NUM_TEST_SHARDS: 1 2022-03-17T19:13:27.4327123Z PR_BODY: 2022-03-17T19:13:27.4327361Z GPU_FLAG: --gpus all 2022-03-17T19:13:27.4327586Z SHM_SIZE: 2g 2022-03-17T19:13:27.4327816Z ##[endgroup] 2022-03-17T19:13:27.4463393Z Filesystem Size Used Avail Use% Mounted on 2022-03-17T19:13:27.4463701Z devtmpfs 129G 0 129G 0% /dev 2022-03-17T19:13:27.4463967Z tmpfs 129G 0 129G 0% /dev/shm 2022-03-17T19:13:27.4464245Z tmpfs 129G 619k 129G 1% /run 2022-03-17T19:13:27.4464515Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2022-03-17T19:13:27.4467923Z /dev/xvda1 162G 25G 138G 15% / 2022-03-17T19:13:27.4468324Z tmpfs 26G 0 26G 0% /run/user/0 2022-03-17T19:13:27.5661635Z ##[group]Run ./.github/scripts/parse_ref.py 2022-03-17T19:13:27.5662017Z ./.github/scripts/parse_ref.py 2022-03-17T19:13:27.5674734Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T19:13:27.5675032Z env: 2022-03-17T19:13:27.5675331Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:13:27.5675841Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:13:27.5676341Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:13:27.5676752Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:13:27.5677120Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:13:27.5677350Z IN_CI: 1 2022-03-17T19:13:27.5677576Z IS_GHA: 1 2022-03-17T19:13:27.5677807Z IN_WHEEL_TEST: 1 2022-03-17T19:13:27.5678097Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:13:27.5678524Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:13:27.5678838Z PR_LABELS: [] 2022-03-17T19:13:27.5679247Z GITHUB_TOKEN: *** 2022-03-17T19:13:27.5679501Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:13:27.5679829Z PR_NUMBER: 2022-03-17T19:13:27.5680042Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:13:27.5680453Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:13:27.5680726Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:13:27.5681214Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:13:27.5681767Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:13:27.5682092Z TEST_CONFIG: distributed 2022-03-17T19:13:27.5682331Z SHARD_NUMBER: 1 2022-03-17T19:13:27.5682618Z NUM_TEST_SHARDS: 1 2022-03-17T19:13:27.5682803Z PR_BODY: 2022-03-17T19:13:27.5683046Z GPU_FLAG: --gpus all 2022-03-17T19:13:27.5683271Z SHM_SIZE: 2g 2022-03-17T19:13:27.5683514Z ##[endgroup] 2022-03-17T19:13:31.7068245Z ##[group]Run set -x 2022-03-17T19:13:31.7068622Z set -x 2022-03-17T19:13:31.7068853Z  2022-03-17T19:13:31.7069109Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2022-03-17T19:13:31.7069461Z  TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh 2022-03-17T19:13:31.7069808Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2022-03-17T19:13:31.7070128Z  TEST_COMMAND=.jenkins/caffe2/test.sh 2022-03-17T19:13:31.7070397Z else 2022-03-17T19:13:31.7070658Z  TEST_COMMAND=.jenkins/pytorch/test.sh 2022-03-17T19:13:31.7070930Z fi 2022-03-17T19:13:31.7071142Z PROXY_ENV= 2022-03-17T19:13:31.7071520Z # NOTE: XLA multiprocessing tests appear to have issues with squid proxy, going to disable for now 2022-03-17T19:13:31.7072000Z # We should investigate whether or not there's a list of hostnames we can add to no_proxy to 2022-03-17T19:13:31.7072463Z # make it so that we shouldn't have to fully disable squid for XLA tests 2022-03-17T19:13:31.7072800Z if [[ $TEST_CONFIG != 'xla' ]]; then 2022-03-17T19:13:31.7073094Z  # shellcheck disable=SC2089 2022-03-17T19:13:31.7074024Z  PROXY_ENV="-e http_proxy=http://internal-tf-lb-20210727220640487900000002-835786077.us-east-1.elb.amazonaws.com:3128 -e https_proxy=http://internal-tf-lb-20210727220640487900000002-835786077.us-east-1.elb.amazonaws.com:3128 -e no_proxy=localhost,127.0.0.1,github.com,amazonaws.com,s3.amazonaws.com,169.254.169.254,169.254.170.2,/var/run/docker.sock" 2022-03-17T19:13:31.7074895Z fi 2022-03-17T19:13:31.7075196Z # detached container should get cleaned up by teardown_ec2_linux 2022-03-17T19:13:31.7075599Z # TODO: Stop building test binaries as part of the build phase 2022-03-17T19:13:31.7075955Z # Used for GPU_FLAG since that doesn't play nice 2022-03-17T19:13:31.7076285Z # shellcheck disable=SC2086,SC2090 2022-03-17T19:13:31.7076590Z container_name=$(docker run \ 2022-03-17T19:13:31.7076846Z  ${GPU_FLAG:-} \ 2022-03-17T19:13:31.7077121Z  -e BUILD_ENVIRONMENT \ 2022-03-17T19:13:31.7077376Z  -e PR_NUMBER \ 2022-03-17T19:13:31.7077670Z  -e CUSTOM_TEST_ARTIFACT_BUILD_DIR \ 2022-03-17T19:13:31.7077949Z  -e GITHUB_ACTIONS \ 2022-03-17T19:13:31.7078207Z  -e IN_CI \ 2022-03-17T19:13:31.7078449Z  -e IS_GHA \ 2022-03-17T19:13:31.7078673Z  -e BRANCH \ 2022-03-17T19:13:31.7078908Z  -e SHA1 \ 2022-03-17T19:13:31.7079152Z  -e AWS_DEFAULT_REGION \ 2022-03-17T19:13:31.7079424Z  -e IN_WHEEL_TEST \ 2022-03-17T19:13:31.7079669Z  -e SHARD_NUMBER \ 2022-03-17T19:13:31.7079932Z  -e JOB_BASE_NAME \ 2022-03-17T19:13:31.7080195Z  -e TEST_CONFIG \ 2022-03-17T19:13:31.7080447Z  -e NUM_TEST_SHARDS \ 2022-03-17T19:13:31.7080709Z  -e PR_BODY \ 2022-03-17T19:13:31.7080976Z  -e PYTORCH_RETRY_TEST_CASES \ 2022-03-17T19:13:31.7081254Z  -e PR_LABELS \ 2022-03-17T19:13:31.7081528Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2022-03-17T19:13:31.7081819Z  -e SCCACHE_BUCKET \ 2022-03-17T19:13:31.7082238Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2022-03-17T19:13:31.7082505Z  ${PROXY_ENV} \ 2022-03-17T19:13:31.7082820Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2022-03-17T19:13:31.7083126Z  --ulimit stack=10485760:83886080 \ 2022-03-17T19:13:31.7083438Z  --security-opt seccomp=unconfined \ 2022-03-17T19:13:31.7083746Z  --cap-add=SYS_PTRACE \ 2022-03-17T19:13:31.7083995Z  --ipc=host \ 2022-03-17T19:13:31.7084261Z  --shm-size="${SHM_SIZE}" \ 2022-03-17T19:13:31.7084502Z  --tty \ 2022-03-17T19:13:31.7084744Z  --detach \ 2022-03-17T19:13:31.7085000Z  --name="${container_name}" \ 2022-03-17T19:13:31.7085342Z  --user jenkins \ 2022-03-17T19:13:31.7085677Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2022-03-17T19:13:31.7086003Z  -w /var/lib/jenkins/workspace \ 2022-03-17T19:13:31.7086283Z  "${DOCKER_IMAGE}" 2022-03-17T19:13:31.7086507Z ) 2022-03-17T19:13:31.7086885Z docker exec -t "${container_name}" sh -c "sudo chown -R jenkins . && pip install dist/*.whl && ${TEST_COMMAND}" 2022-03-17T19:13:31.7100193Z shell: /usr/bin/bash -e {0} 2022-03-17T19:13:31.7100708Z env: 2022-03-17T19:13:31.7101000Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:13:31.7101512Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T19:13:31.7102014Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T19:13:31.7102424Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T19:13:31.7102792Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T19:13:31.7103021Z IN_CI: 1 2022-03-17T19:13:31.7103238Z IS_GHA: 1 2022-03-17T19:13:31.7103450Z IN_WHEEL_TEST: 1 2022-03-17T19:13:31.7103756Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T19:13:31.7104146Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T19:13:31.7104459Z PR_LABELS: [] 2022-03-17T19:13:31.7104883Z GITHUB_TOKEN: *** 2022-03-17T19:13:31.7105152Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T19:13:31.7105390Z PR_NUMBER: 2022-03-17T19:13:31.7105666Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T19:13:31.7105939Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T19:13:31.7106211Z GIT_DEFAULT_BRANCH: master 2022-03-17T19:13:31.7106708Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:13:31.7107266Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T19:13:31.7107595Z TEST_CONFIG: distributed 2022-03-17T19:13:31.7107829Z SHARD_NUMBER: 1 2022-03-17T19:13:31.7108066Z NUM_TEST_SHARDS: 1 2022-03-17T19:13:31.7108286Z PR_BODY: 2022-03-17T19:13:31.7108522Z GPU_FLAG: --gpus all 2022-03-17T19:13:31.7108758Z SHM_SIZE: 2g 2022-03-17T19:13:31.7108983Z BRANCH: master 2022-03-17T19:13:31.7109218Z ##[endgroup] 2022-03-17T19:13:31.7137196Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2022-03-17T19:13:31.7137643Z + [[ linux-xenial-cuda11.3-py3.7-gcc7 == *onnx* ]] 2022-03-17T19:13:31.7137968Z + TEST_COMMAND=.jenkins/pytorch/test.sh 2022-03-17T19:13:31.7138238Z + PROXY_ENV= 2022-03-17T19:13:31.7138469Z + [[ distributed != \x\l\a ]] 2022-03-17T19:13:31.7139677Z + PROXY_ENV='-e http_proxy=http://internal-tf-lb-20210727220640487900000002-835786077.us-east-1.elb.amazonaws.com:3128 -e https_proxy=http://internal-tf-lb-20210727220640487900000002-835786077.us-east-1.elb.amazonaws.com:3128 -e no_proxy=localhost,127.0.0.1,github.com,amazonaws.com,s3.amazonaws.com,169.254.169.254,169.254.170.2,/var/run/docker.sock' 2022-03-17T19:13:31.7148695Z +++ nproc --ignore=2 2022-03-17T19:13:31.7179171Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e CUSTOM_TEST_ARTIFACT_BUILD_DIR -e GITHUB_ACTIONS -e IN_CI -e IS_GHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e JOB_BASE_NAME -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e PYTORCH_RETRY_TEST_CASES -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e http_proxy=http://internal-tf-lb-20210727220640487900000002-835786077.us-east-1.elb.amazonaws.com:3128 -e https_proxy=http://internal-tf-lb-20210727220640487900000002-835786077.us-east-1.elb.amazonaws.com:3128 -e no_proxy=localhost,127.0.0.1,github.com,amazonaws.com,s3.amazonaws.com,169.254.169.254,169.254.170.2,/var/run/docker.sock --env-file=/tmp/github_env_2000264336 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T19:13:55.7536427Z + container_name=68188c1d65840e77017513044fc8915cd31339d136d8cd4b0d622137ce388559 2022-03-17T19:13:55.7537313Z + docker exec -t 68188c1d65840e77017513044fc8915cd31339d136d8cd4b0d622137ce388559 sh -c 'sudo chown -R jenkins . && pip install dist/*.whl && .jenkins/pytorch/test.sh' 2022-03-17T19:13:56.7081614Z Processing ./dist/torch-1.12.0a0+git4b4f652-cp37-cp37m-linux_x86_64.whl 2022-03-17T19:13:56.8011026Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.12.0a0+git4b4f652) (3.10.0.2) 2022-03-17T19:13:57.2986053Z Installing collected packages: torch 2022-03-17T19:14:08.6735087Z Successfully installed torch-1.12.0a0+git4b4f652 2022-03-17T19:14:08.7231539Z + COMPACT_JOB_NAME=linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T19:14:08.7234527Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2022-03-17T19:14:08.7449143Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch 2022-03-17T19:14:08.7449913Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin 2022-03-17T19:14:08.7450384Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib 2022-03-17T19:14:08.7451325Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test 2022-03-17T19:14:08.7451659Z + BUILD_DIR=build 2022-03-17T19:14:08.7454098Z + BUILD_RENAMED_DIR=build_renamed 2022-03-17T19:14:08.7454394Z + BUILD_BIN_DIR=build/bin 2022-03-17T19:14:08.7455039Z + [[ -n distributed ]] 2022-03-17T19:14:08.7455608Z + BUILD_ENVIRONMENT=linux-xenial-cuda11.3-py3.7-gcc7-distributed 2022-03-17T19:14:08.7456203Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *bazel* ]] 2022-03-17T19:14:08.7456599Z ++ realpath build/custom_test_artifacts 2022-03-17T19:14:08.7462021Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2022-03-17T19:14:08.7466526Z ++ dirname .jenkins/pytorch/test.sh 2022-03-17T19:14:08.7472804Z + source .jenkins/pytorch/common.sh 2022-03-17T19:14:08.7477628Z +++ dirname .jenkins/pytorch/common.sh 2022-03-17T19:14:08.7488244Z ++ source .jenkins/pytorch/common_utils.sh 2022-03-17T19:14:08.7491561Z +++ TORCHVISION_COMMIT=8a2dc6f22ac4389ccba8859aa1e1cb14f1ee53db 2022-03-17T19:14:08.7492241Z ++ set -ex 2022-03-17T19:14:08.7500004Z ++++ dirname .jenkins/pytorch/common.sh 2022-03-17T19:14:08.7511257Z +++ cd .jenkins/pytorch 2022-03-17T19:14:08.7511584Z +++ pwd -P 2022-03-17T19:14:08.7514661Z ++ SCRIPT_DIR=/var/lib/jenkins/workspace/.jenkins/pytorch 2022-03-17T19:14:08.7515207Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *rocm* ]] 2022-03-17T19:14:08.7515565Z ++ echo ENTERED_USER_LAND 2022-03-17T19:14:08.7515872Z ENTERED_USER_LAND 2022-03-17T19:14:08.7516137Z ++ export IN_CI=1 2022-03-17T19:14:08.7516502Z ++ IN_CI=1 2022-03-17T19:14:08.7517446Z ++ declare -f -t trap_add 2022-03-17T19:14:08.7517712Z ++ trap_add cleanup EXIT 2022-03-17T19:14:08.7518027Z ++ trap_add_cmd=cleanup 2022-03-17T19:14:08.7518262Z ++ shift 2022-03-17T19:14:08.7518699Z ++ for trap_add_name in '"$@"' 2022-03-17T19:14:08.7526896Z ++++ trap -p EXIT 2022-03-17T19:14:08.7530328Z +++ eval 'extract_trap_cmd ' 2022-03-17T19:14:08.7530673Z ++++ extract_trap_cmd 2022-03-17T19:14:08.7530978Z ++++ printf '%s\n' '' 2022-03-17T19:14:08.7531405Z +++ printf '%s\n' cleanup 2022-03-17T19:14:08.7533659Z ++ trap -- ' 2022-03-17T19:14:08.7533951Z cleanup' EXIT 2022-03-17T19:14:08.7536243Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *win-* ]] 2022-03-17T19:14:08.7536599Z ++ which sccache 2022-03-17T19:14:08.7547587Z ++ sccache --stop-server 2022-03-17T19:14:08.7572342Z ++ true 2022-03-17T19:14:08.7572726Z ++ rm -f /var/lib/jenkins/sccache_error.log 2022-03-17T19:14:08.7581569Z ++ [[ -n '' ]] 2022-03-17T19:14:08.7582013Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *rocm* ]] 2022-03-17T19:14:08.7582685Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2022-03-17T19:14:08.7583029Z ++ SCCACHE_IDLE_TIMEOUT=1200 2022-03-17T19:14:08.7583406Z ++ RUST_LOG=sccache::server=error 2022-03-17T19:14:08.7583790Z ++ sccache --start-server 2022-03-17T19:14:08.7600769Z sccache: Starting the server... 2022-03-17T19:14:08.7866888Z ++ sccache --zero-stats 2022-03-17T19:14:08.7888718Z Compile requests 0 2022-03-17T19:14:08.7889101Z Compile requests executed 0 2022-03-17T19:14:08.7889450Z Cache hits 0 2022-03-17T19:14:08.7890616Z Cache misses 0 2022-03-17T19:14:08.7891420Z Cache timeouts 0 2022-03-17T19:14:08.7891720Z Cache read errors 0 2022-03-17T19:14:08.7911911Z Forced recaches 0 2022-03-17T19:14:08.7912241Z Cache write errors 0 2022-03-17T19:14:08.7912564Z Compilation failures 0 2022-03-17T19:14:08.7912855Z Cache errors 0 2022-03-17T19:14:08.7913214Z Non-cacheable compilations 0 2022-03-17T19:14:08.7913560Z Non-cacheable calls 0 2022-03-17T19:14:08.7913886Z Non-compilation calls 0 2022-03-17T19:14:08.7914185Z Unsupported compiler calls 0 2022-03-17T19:14:08.7914494Z Average cache write 0.000 s 2022-03-17T19:14:08.7914788Z Average cache read miss 0.000 s 2022-03-17T19:14:08.7915198Z Average cache read hit 0.000 s 2022-03-17T19:14:08.7915669Z Failed distributed compilations 0 2022-03-17T19:14:08.7917015Z Cache location S3, bucket: Bucket(name=ossci-compiler-cache-circleci-v2, base_url=http://ossci-compiler-cache-circleci-v2.s3.amazonaws.com/) 2022-03-17T19:14:08.7918277Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-test == *-build ]] 2022-03-17T19:14:08.7918876Z ++ which ccache 2022-03-17T19:14:08.7919604Z ++ '[' -z linux-xenial-cuda11.3-py3.7-gcc7 ']' 2022-03-17T19:14:08.7920639Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *linux-trusty-py3.6-gcc7* ]] 2022-03-17T19:14:08.7921420Z ++ BUILD_TEST_LIBTORCH=0 2022-03-17T19:14:08.7921940Z ++ [[ distributed == *xla* ]] 2022-03-17T19:14:08.7922779Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *centos* ]] 2022-03-17T19:14:08.7923820Z ++ [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *linux-bionic* ]] 2022-03-17T19:14:08.7924599Z + echo 'Testing pytorch' 2022-03-17T19:14:08.7925109Z Testing pytorch 2022-03-17T19:14:08.7925639Z + export LANG=C.UTF-8 2022-03-17T19:14:08.7926107Z + LANG=C.UTF-8 2022-03-17T19:14:08.7926521Z + PR_NUMBER= 2022-03-17T19:14:08.7926978Z + [[ distributed == \d\e\f\a\u\l\t ]] 2022-03-17T19:14:08.7927810Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-slow-* ]] 2022-03-17T19:14:08.7928471Z + [[ distributed == \s\l\o\w ]] 2022-03-17T19:14:08.7929364Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *slow-gradcheck* ]] 2022-03-17T19:14:08.7930426Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *cuda* ]] 2022-03-17T19:14:08.7931201Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-03-17T19:14:08.7931841Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2022-03-17T19:14:08.7932675Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *cuda11* ]] 2022-03-17T19:14:08.7933386Z + export BUILD_SPLIT_CUDA=ON 2022-03-17T19:14:08.7934076Z + BUILD_SPLIT_CUDA=ON 2022-03-17T19:14:08.7934902Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *noarch* ]] 2022-03-17T19:14:08.7935577Z + export PYTORCH_TEST_SKIP_NOARCH=1 2022-03-17T19:14:08.7936114Z + PYTORCH_TEST_SKIP_NOARCH=1 2022-03-17T19:14:08.7936662Z + [[ -n '' ]] 2022-03-17T19:14:08.7937197Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0 2022-03-17T19:14:08.7937824Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0 2022-03-17T19:14:08.7938673Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *rocm* ]] 2022-03-17T19:14:08.7939661Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *ppc64le* ]] 2022-03-17T19:14:08.7941164Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *-bazel-* ]] 2022-03-17T19:14:08.7941933Z + pip_install --user ninja 2022-03-17T19:14:08.7942629Z + pip install --progress-bar off --user ninja 2022-03-17T19:14:09.3748344Z Collecting ninja 2022-03-17T19:14:09.4068187Z Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2022-03-17T19:14:09.4162794Z [?25l 2022-03-17T19:14:09.8451331Z [?25hInstalling collected packages: ninja 2022-03-17T19:14:09.8566126Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2022-03-17T19:14:09.8566745Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-03-17T19:14:09.8624570Z Successfully installed ninja-1.10.2.3 2022-03-17T19:14:09.9137823Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-03-17T19:14:09.9138530Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2022-03-17T19:14:09.9139831Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *asan* ]] 2022-03-17T19:14:09.9140595Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-NO_AVX-* ]] 2022-03-17T19:14:09.9141003Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X ]] 2022-03-17T19:14:09.9141474Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-NO_AVX2-* ]] 2022-03-17T19:14:09.9141812Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2022-03-17T19:14:09.9142272Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-NO_AVX512-* ]] 2022-03-17T19:14:09.9142645Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\5\1\2 ]] 2022-03-17T19:14:09.9145752Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *tbb* ]] 2022-03-17T19:14:09.9159064Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *libtorch* ]] 2022-03-17T19:14:09.9159594Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-bazel-* ]] 2022-03-17T19:14:09.9162443Z + cd test 2022-03-17T19:14:09.9163284Z + python -c 'import torch; print(torch.__config__.show())' 2022-03-17T19:14:14.3586924Z PyTorch built with: 2022-03-17T19:14:14.3587413Z - GCC 7.5 2022-03-17T19:14:14.3587721Z - C++ Version: 201402 2022-03-17T19:14:14.3588456Z - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-03-17T19:14:14.3589288Z - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e) 2022-03-17T19:14:14.3589927Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-03-17T19:14:14.3590490Z - LAPACK is enabled (usually provided by MKL) 2022-03-17T19:14:14.3591025Z - NNPACK is enabled 2022-03-17T19:14:14.3591559Z - CPU capability usage: AVX2 2022-03-17T19:14:14.3592058Z - CUDA Runtime 11.3 2022-03-17T19:14:14.3592703Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2022-03-17T19:14:14.3593264Z - CuDNN 8.2 2022-03-17T19:14:14.3593696Z - Magma 2.5.2 2022-03-17T19:14:14.3597319Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 2022-03-17T19:14:14.3599568Z 2022-03-17T19:14:14.9322114Z + cd test 2022-03-17T19:14:14.9322695Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2022-03-17T19:14:15.5861606Z ATen/Parallel: 2022-03-17T19:14:15.5862062Z at::get_num_threads() : 16 2022-03-17T19:14:15.5862559Z at::get_num_interop_threads() : 16 2022-03-17T19:14:15.5862869Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2022-03-17T19:14:15.5863080Z omp_get_max_threads() : 16 2022-03-17T19:14:15.5863711Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications 2022-03-17T19:14:15.5864100Z mkl_get_max_threads() : 16 2022-03-17T19:14:15.5864541Z Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e) 2022-03-17T19:14:15.5864899Z std::thread::hardware_concurrency() : 32 2022-03-17T19:14:15.5865170Z Environment variables: 2022-03-17T19:14:15.5865434Z OMP_NUM_THREADS : [not set] 2022-03-17T19:14:15.5865681Z MKL_NUM_THREADS : [not set] 2022-03-17T19:14:15.5865955Z ATen parallel backend: OpenMP 2022-03-17T19:14:15.5866142Z 2022-03-17T19:14:15.6703023Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *deploy* ]] 2022-03-17T19:14:15.6703820Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *backward* ]] 2022-03-17T19:14:15.6704201Z + [[ distributed == *xla* ]] 2022-03-17T19:14:15.6704619Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *jit_legacy-test ]] 2022-03-17T19:14:15.6705114Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-test == *jit_legacy-test ]] 2022-03-17T19:14:15.6705473Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2022-03-17T19:14:15.6705902Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *libtorch* ]] 2022-03-17T19:14:15.6706421Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-test1 ]] 2022-03-17T19:14:15.6706898Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-test == *-test1 ]] 2022-03-17T19:14:15.6707182Z + [[ 1 == 1 ]] 2022-03-17T19:14:15.6707440Z + [[ 1 -gt 1 ]] 2022-03-17T19:14:15.6707820Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-test2 ]] 2022-03-17T19:14:15.6708294Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-test == *-test2 ]] 2022-03-17T19:14:15.6708570Z + [[ 1 == 2 ]] 2022-03-17T19:14:15.6708822Z + [[ 1 -gt 2 ]] 2022-03-17T19:14:15.6709202Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *vulkan* ]] 2022-03-17T19:14:15.6709699Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *-bazel-* ]] 2022-03-17T19:14:15.6710211Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *distributed* ]] 2022-03-17T19:14:15.6710541Z + test_distributed 2022-03-17T19:14:15.6710877Z + echo 'Testing distributed python tests' 2022-03-17T19:14:15.6711165Z Testing distributed python tests 2022-03-17T19:14:15.6711561Z + python test/run_test.py --distributed-tests --verbose 2022-03-17T19:14:21.8158881Z Ignoring disabled issues: [] 2022-03-17T19:14:21.8280564Z Selected tests: 2022-03-17T19:14:21.8280956Z distributed/_shard/sharded_optim/test_sharded_optim 2022-03-17T19:14:21.8281282Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 2022-03-17T19:14:21.8281987Z distributed/_shard/sharded_tensor/ops/test_elementwise_ops 2022-03-17T19:14:21.8282295Z distributed/_shard/sharded_tensor/ops/test_embedding 2022-03-17T19:14:21.8282648Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 2022-03-17T19:14:21.8283062Z distributed/_shard/sharded_tensor/ops/test_init 2022-03-17T19:14:21.8283383Z distributed/_shard/sharded_tensor/ops/test_linear 2022-03-17T19:14:21.8283743Z distributed/_shard/sharded_tensor/test_megatron_prototype 2022-03-17T19:14:21.8284020Z distributed/_shard/sharded_tensor/test_partial_tensor 2022-03-17T19:14:21.8284349Z distributed/_shard/sharded_tensor/test_sharded_tensor 2022-03-17T19:14:21.8284708Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 2022-03-17T19:14:21.8285157Z distributed/_shard/sharding_spec/test_sharding_spec 2022-03-17T19:14:21.8285494Z distributed/algorithms/test_join 2022-03-17T19:14:21.8285782Z distributed/elastic/events/lib_test 2022-03-17T19:14:21.8286078Z distributed/elastic/metrics/api_test 2022-03-17T19:14:21.8286410Z distributed/elastic/multiprocessing/api_test 2022-03-17T19:14:21.8286709Z distributed/elastic/timer/api_test 2022-03-17T19:14:21.8287061Z distributed/elastic/timer/local_timer_example 2022-03-17T19:14:21.8287335Z distributed/elastic/timer/local_timer_test 2022-03-17T19:14:21.8287656Z distributed/elastic/utils/distributed_test 2022-03-17T19:14:21.8287949Z distributed/elastic/utils/logging_test 2022-03-17T19:14:21.8288279Z distributed/elastic/utils/util_test 2022-03-17T19:14:21.8288567Z distributed/fsdp/test_flatten_params_wrapper 2022-03-17T19:14:21.8288858Z distributed/fsdp/test_fsdp_apply 2022-03-17T19:14:21.8289198Z distributed/fsdp/test_fsdp_checkpoint 2022-03-17T19:14:21.8289526Z distributed/fsdp/test_fsdp_clip_grad_norm 2022-03-17T19:14:21.8289810Z distributed/fsdp/test_fsdp_comm 2022-03-17T19:14:21.8290095Z distributed/fsdp/test_fsdp_core 2022-03-17T19:14:21.8290394Z distributed/fsdp/test_fsdp_freezing_weights 2022-03-17T19:14:21.8290742Z distributed/fsdp/test_fsdp_grad_acc 2022-03-17T19:14:21.8290990Z distributed/fsdp/test_fsdp_input 2022-03-17T19:14:21.8291281Z distributed/fsdp/test_fsdp_memory 2022-03-17T19:14:21.8291593Z distributed/fsdp/test_fsdp_multiple_forward 2022-03-17T19:14:21.8291920Z distributed/fsdp/test_fsdp_multiple_wrapping 2022-03-17T19:14:21.8292223Z distributed/fsdp/test_fsdp_overlap 2022-03-17T19:14:21.8292506Z distributed/fsdp/test_fsdp_pure_fp16 2022-03-17T19:14:21.8292814Z distributed/fsdp/test_fsdp_state_dict 2022-03-17T19:14:21.8293150Z distributed/fsdp/test_fsdp_summon_full_params 2022-03-17T19:14:21.8293431Z distributed/fsdp/test_fsdp_traversal 2022-03-17T19:14:21.8293726Z distributed/fsdp/test_fsdp_uneven 2022-03-17T19:14:21.8293993Z distributed/fsdp/test_utils 2022-03-17T19:14:21.8294305Z distributed/fsdp/test_wrap 2022-03-17T19:14:21.8294538Z distributed/nn/jit/test_instantiator 2022-03-17T19:14:21.8294860Z distributed/optim/test_zero_redundancy_optimizer 2022-03-17T19:14:21.8295178Z distributed/pipeline/sync/skip/test_api 2022-03-17T19:14:21.8295538Z distributed/pipeline/sync/skip/test_gpipe 2022-03-17T19:14:21.8295902Z distributed/pipeline/sync/skip/test_inspect_skip_layout 2022-03-17T19:14:21.8296230Z distributed/pipeline/sync/skip/test_leak 2022-03-17T19:14:21.8296552Z distributed/pipeline/sync/skip/test_portal 2022-03-17T19:14:21.8296898Z distributed/pipeline/sync/skip/test_stash_pop 2022-03-17T19:14:21.8297207Z distributed/pipeline/sync/skip/test_tracker 2022-03-17T19:14:21.8297528Z distributed/pipeline/sync/skip/test_verify_skippables 2022-03-17T19:14:21.8297860Z distributed/pipeline/sync/test_balance 2022-03-17T19:14:21.8298172Z distributed/pipeline/sync/test_bugs 2022-03-17T19:14:21.8298463Z distributed/pipeline/sync/test_checkpoint 2022-03-17T19:14:21.8298778Z distributed/pipeline/sync/test_copy 2022-03-17T19:14:21.8299088Z distributed/pipeline/sync/test_deferred_batch_norm 2022-03-17T19:14:21.8299423Z distributed/pipeline/sync/test_dependency 2022-03-17T19:14:21.8299717Z distributed/pipeline/sync/test_inplace 2022-03-17T19:14:21.8300118Z distributed/pipeline/sync/test_microbatch 2022-03-17T19:14:21.8300770Z distributed/pipeline/sync/test_phony 2022-03-17T19:14:21.8301060Z distributed/pipeline/sync/test_pipe 2022-03-17T19:14:21.8301370Z distributed/pipeline/sync/test_pipeline 2022-03-17T19:14:21.8301668Z distributed/pipeline/sync/test_stream 2022-03-17T19:14:21.8301989Z distributed/pipeline/sync/test_transparency 2022-03-17T19:14:21.8302290Z distributed/pipeline/sync/test_worker 2022-03-17T19:14:21.8302609Z distributed/rpc/cuda/test_tensorpipe_agent 2022-03-17T19:14:21.8302953Z distributed/rpc/test_faulty_agent 2022-03-17T19:14:21.8303199Z distributed/rpc/test_tensorpipe_agent 2022-03-17T19:14:21.8303586Z distributed/test_c10d_common 2022-03-17T19:14:21.8303853Z distributed/test_c10d_gloo 2022-03-17T19:14:21.8304120Z distributed/test_c10d_nccl 2022-03-17T19:14:21.8304380Z distributed/test_c10d_spawn_gloo 2022-03-17T19:14:21.8304662Z distributed/test_c10d_spawn_nccl 2022-03-17T19:14:21.8304944Z distributed/test_data_parallel 2022-03-17T19:14:21.8305223Z distributed/test_distributed_spawn 2022-03-17T19:14:21.8305543Z distributed/test_launcher 2022-03-17T19:14:21.8305753Z distributed/test_nccl 2022-03-17T19:14:21.8306020Z distributed/test_pg_wrapper 2022-03-17T19:14:21.8306270Z distributed/test_store 2022-03-17T19:14:21.8387190Z Prioritized test from test file changes. 2022-03-17T19:14:21.8387537Z reordering tests for PR: 2022-03-17T19:14:21.8387779Z prioritized: [] 2022-03-17T19:14:21.8394922Z the rest: ['distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/_shard/sharded_tensor/test_partial_tensor', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/algorithms/test_join', 'distributed/elastic/events/lib_test', 'distributed/elastic/metrics/api_test', 'distributed/elastic/multiprocessing/api_test', 'distributed/elastic/timer/api_test', 'distributed/elastic/timer/local_timer_example', 'distributed/elastic/timer/local_timer_test', 'distributed/elastic/utils/distributed_test', 'distributed/elastic/utils/logging_test', 'distributed/elastic/utils/util_test', 'distributed/fsdp/test_flatten_params_wrapper', 'distributed/fsdp/test_fsdp_apply', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_core', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_input', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_utils', 'distributed/fsdp/test_wrap', 'distributed/nn/jit/test_instantiator', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_worker', 'distributed/rpc/cuda/test_tensorpipe_agent', 'distributed/rpc/test_faulty_agent', 'distributed/rpc/test_tensorpipe_agent', 'distributed/test_c10d_common', 'distributed/test_c10d_gloo', 'distributed/test_c10d_nccl', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_spawn_nccl', 'distributed/test_data_parallel', 'distributed/test_distributed_spawn', 'distributed/test_launcher', 'distributed/test_nccl', 'distributed/test_pg_wrapper', 'distributed/test_store'] 2022-03-17T19:14:21.8399654Z 2022-03-17T19:14:21.9222223Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-03-17 19:14:21.921842] 2022-03-17T19:14:21.9222970Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:14:21.921907] 2022-03-17T19:14:22.7411709Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim 2022-03-17T19:14:22.7429325Z 2022-03-17T19:14:22.7429538Z Running tests... 2022-03-17T19:14:22.7430018Z ---------------------------------------------------------------------- 2022-03-17T19:14:24.2788932Z test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:14:24.3156690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 276 2022-03-17T19:14:24.3257040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 277 2022-03-17T19:14:24.3358388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 278 2022-03-17T19:14:24.3460023Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 279 2022-03-17T19:14:25.1678782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:25.1884430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:25.2028778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:25.2049849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:25.3498985Z skip (2.607s) 2022-03-17T19:14:25.3631604Z test_sharded_optim (__main__.TestShardedOptimizer) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 420 2022-03-17T19:14:25.3732272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 421 2022-03-17T19:14:25.3833425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 422 2022-03-17T19:14:25.3959709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 423 2022-03-17T19:14:26.2636093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:26.2641107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:26.2650189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:26.2662818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:26.3995114Z skip (1.049s) 2022-03-17T19:14:26.3995344Z 2022-03-17T19:14:26.3995781Z ---------------------------------------------------------------------- 2022-03-17T19:14:26.3996183Z Ran 2 tests in 3.657s 2022-03-17T19:14:26.3996322Z 2022-03-17T19:14:26.3996421Z OK (skipped=2) 2022-03-17T19:14:26.3996547Z 2022-03-17T19:14:26.3996674Z Generating XML reports... 2022-03-17T19:14:26.4043498Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220317191422.xml 2022-03-17T19:14:26.6341716Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-03-17 19:14:26.633653] 2022-03-17T19:14:26.6342501Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:14:26.633756] 2022-03-17T19:14:27.4274555Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp 2022-03-17T19:14:27.4291488Z 2022-03-17T19:14:27.4291666Z Running tests... 2022-03-17T19:14:27.4292300Z ---------------------------------------------------------------------- 2022-03-17T19:14:27.4304082Z test_torch_allclose (__main__.TestShardedTensorBinaryOps) 2022-03-17T19:14:28.9632254Z Test torch.allclose(ShardedTensor, ShardedTensor) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:14:29.0002926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 601 2022-03-17T19:14:29.0104973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 602 2022-03-17T19:14:29.0207711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 603 2022-03-17T19:14:29.0311167Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 604 2022-03-17T19:14:29.8478110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:29.8481117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:29.8762332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:29.8932417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:30.0349674Z skip (2.605s) 2022-03-17T19:14:30.0467768Z test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 745 2022-03-17T19:14:30.0569540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 746 2022-03-17T19:14:30.0672239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 747 2022-03-17T19:14:30.0798213Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 748 2022-03-17T19:14:30.8929035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:30.9068680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:30.9199341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:30.9246847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:31.0832176Z skip (1.048s) 2022-03-17T19:14:31.0838851Z test_torch_equal (__main__.TestShardedTensorBinaryOps) 2022-03-17T19:14:31.0948762Z Test torch.equal(ShardedTensor, ShardedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 889 2022-03-17T19:14:31.1050877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 890 2022-03-17T19:14:31.1152809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 891 2022-03-17T19:14:31.1255359Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 892 2022-03-17T19:14:31.9590331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:31.9672652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:31.9841591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:31.9932821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:32.1289123Z skip (1.046s) 2022-03-17T19:14:32.1415042Z test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1033 2022-03-17T19:14:32.1520900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1034 2022-03-17T19:14:32.1624113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1035 2022-03-17T19:14:32.1735726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1036 2022-03-17T19:14:33.0243299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:33.0403001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:33.0551426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:33.0689373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:33.2771057Z skip (1.148s) 2022-03-17T19:14:33.2771253Z 2022-03-17T19:14:33.2771925Z ---------------------------------------------------------------------- 2022-03-17T19:14:33.2772247Z Ran 4 tests in 5.848s 2022-03-17T19:14:33.2772348Z 2022-03-17T19:14:33.2772465Z OK (skipped=4) 2022-03-17T19:14:33.2772622Z 2022-03-17T19:14:33.2772751Z Generating XML reports... 2022-03-17T19:14:33.2822532Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220317191427.xml 2022-03-17T19:14:33.5139841Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-03-17 19:14:33.513480] 2022-03-17T19:14:33.5140890Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:14:33.513582] 2022-03-17T19:14:34.2677826Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops 2022-03-17T19:14:34.2695079Z 2022-03-17T19:14:34.2695435Z Running tests... 2022-03-17T19:14:34.2695818Z ---------------------------------------------------------------------- 2022-03-17T19:14:35.8045312Z test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:14:35.8419669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1214 2022-03-17T19:14:35.8547530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1215 2022-03-17T19:14:35.8651599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1216 2022-03-17T19:14:35.8756066Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1217 2022-03-17T19:14:36.6525596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:36.7206518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:36.7235812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:36.7254361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:36.8794434Z skip (2.610s) 2022-03-17T19:14:36.8916978Z test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1358 2022-03-17T19:14:36.9020372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1359 2022-03-17T19:14:36.9123068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1360 2022-03-17T19:14:36.9227775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1361 2022-03-17T19:14:37.7925202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:37.8039367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:37.8060031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:37.8831717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:38.0266088Z skip (1.147s) 2022-03-17T19:14:38.0266305Z 2022-03-17T19:14:38.0266698Z ---------------------------------------------------------------------- 2022-03-17T19:14:38.0267038Z Ran 2 tests in 3.757s 2022-03-17T19:14:38.0267189Z 2022-03-17T19:14:38.0267296Z OK (skipped=2) 2022-03-17T19:14:38.0267444Z 2022-03-17T19:14:38.0267571Z Generating XML reports... 2022-03-17T19:14:38.0313733Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220317191434.xml 2022-03-17T19:14:38.2658370Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-03-17 19:14:38.265316] 2022-03-17T19:14:38.2659217Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:14:38.265417] 2022-03-17T19:14:39.0183508Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding 2022-03-17T19:14:39.0200353Z 2022-03-17T19:14:39.0200648Z Running tests... 2022-03-17T19:14:39.0201025Z ---------------------------------------------------------------------- 2022-03-17T19:14:40.5579038Z test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:14:40.5953680Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1539 2022-03-17T19:14:40.6079344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1540 2022-03-17T19:14:40.6182684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1541 2022-03-17T19:14:40.6289152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1542 2022-03-17T19:14:41.4444329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:41.4826220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:41.5044081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:41.5117755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:41.6326051Z skip (2.612s) 2022-03-17T19:14:41.6459818Z test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1683 2022-03-17T19:14:41.6564558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1684 2022-03-17T19:14:41.6668511Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1685 2022-03-17T19:14:41.6773195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1686 2022-03-17T19:14:42.4698183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:42.4770642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:42.4796434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:42.5223716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:42.6807288Z skip (1.048s) 2022-03-17T19:14:42.6807520Z 2022-03-17T19:14:42.6808149Z ---------------------------------------------------------------------- 2022-03-17T19:14:42.6808499Z Ran 2 tests in 3.661s 2022-03-17T19:14:42.6808645Z 2022-03-17T19:14:42.6808758Z OK (skipped=2) 2022-03-17T19:14:42.6808912Z 2022-03-17T19:14:42.6809054Z Generating XML reports... 2022-03-17T19:14:42.6854361Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220317191439.xml 2022-03-17T19:14:42.9113379Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-03-17 19:14:42.910838] 2022-03-17T19:14:42.9114480Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:14:42.910941] 2022-03-17T19:14:43.7149432Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag 2022-03-17T19:14:43.7166664Z 2022-03-17T19:14:43.7166811Z Running tests... 2022-03-17T19:14:43.7167540Z ---------------------------------------------------------------------- 2022-03-17T19:14:45.2459953Z test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:14:45.2836484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1864 2022-03-17T19:14:45.2963514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1865 2022-03-17T19:14:45.3068535Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1866 2022-03-17T19:14:45.3174262Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1867 2022-03-17T19:14:46.1396338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:46.1557151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:46.1668700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:46.1953019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:46.3211832Z skip (2.604s) 2022-03-17T19:14:46.3338219Z test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2008 2022-03-17T19:14:46.3443382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2009 2022-03-17T19:14:46.3547725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2010 2022-03-17T19:14:46.3653437Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2011 2022-03-17T19:14:47.1770279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:47.2006401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:47.2335961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:47.2966742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:47.4688710Z skip (1.147s) 2022-03-17T19:14:47.4688925Z 2022-03-17T19:14:47.4689301Z ---------------------------------------------------------------------- 2022-03-17T19:14:47.4689641Z Ran 2 tests in 3.752s 2022-03-17T19:14:47.4689776Z 2022-03-17T19:14:47.4689854Z OK (skipped=2) 2022-03-17T19:14:47.4690007Z 2022-03-17T19:14:47.4690132Z Generating XML reports... 2022-03-17T19:14:47.4735013Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220317191443.xml 2022-03-17T19:14:47.7007953Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-03-17 19:14:47.700321] 2022-03-17T19:14:47.7008689Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:14:47.700425] 2022-03-17T19:14:48.4977056Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init 2022-03-17T19:14:48.4994235Z 2022-03-17T19:14:48.4994326Z Running tests... 2022-03-17T19:14:48.4995008Z ---------------------------------------------------------------------- 2022-03-17T19:14:48.5011838Z test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit) 2022-03-17T19:14:50.0283458Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:14:50.0656385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2189 2022-03-17T19:14:50.0781522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2190 2022-03-17T19:14:50.0883979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2191 2022-03-17T19:14:50.0988485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2192 2022-03-17T19:14:50.8941381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:50.9167296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:50.9313257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:50.9441728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:51.1029088Z skip (2.603s) 2022-03-17T19:14:51.1044969Z test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit) 2022-03-17T19:14:51.1170321Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2333 2022-03-17T19:14:51.1275176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2334 2022-03-17T19:14:51.1379618Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2335 2022-03-17T19:14:51.1484102Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2336 2022-03-17T19:14:52.0003570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:52.0297002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:52.0315899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:52.0330515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:52.2519000Z skip (1.149s) 2022-03-17T19:14:52.2534310Z test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit) 2022-03-17T19:14:52.2647319Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2477 2022-03-17T19:14:52.2750896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2478 2022-03-17T19:14:52.2854337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2479 2022-03-17T19:14:52.2957951Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2480 2022-03-17T19:14:53.0997197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:53.1237267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:53.1280666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:53.1348709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:53.2994852Z skip (1.047s) 2022-03-17T19:14:53.2995217Z 2022-03-17T19:14:53.2995614Z ---------------------------------------------------------------------- 2022-03-17T19:14:53.2995940Z Ran 3 tests in 4.800s 2022-03-17T19:14:53.2996149Z 2022-03-17T19:14:53.2996191Z OK (skipped=3) 2022-03-17T19:14:53.2996513Z 2022-03-17T19:14:53.2996771Z Generating XML reports... 2022-03-17T19:14:53.3043144Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220317191448.xml 2022-03-17T19:14:53.5293372Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-03-17 19:14:53.528841] 2022-03-17T19:14:53.5294117Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:14:53.528946] 2022-03-17T19:14:54.3281689Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear 2022-03-17T19:14:54.3298794Z 2022-03-17T19:14:54.3299122Z Running tests... 2022-03-17T19:14:54.3299616Z ---------------------------------------------------------------------- 2022-03-17T19:14:55.8567433Z test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:14:55.8942269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2658 2022-03-17T19:14:55.9045305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2659 2022-03-17T19:14:55.9150629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2660 2022-03-17T19:14:55.9254490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2661 2022-03-17T19:14:56.7203974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:56.8095679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:56.8120898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:56.8125341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:57.0295029Z skip (2.699s) 2022-03-17T19:14:57.0454631Z test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2802 2022-03-17T19:14:57.0558856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2803 2022-03-17T19:14:57.0661346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2804 2022-03-17T19:14:57.0789747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2805 2022-03-17T19:14:57.8604497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:57.8779224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:57.8779712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:57.8788046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:58.0823886Z skip (1.053s) 2022-03-17T19:14:58.0955299Z test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2946 2022-03-17T19:14:58.1059667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2947 2022-03-17T19:14:58.1164418Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2948 2022-03-17T19:14:58.1268873Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2949 2022-03-17T19:14:58.9146036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:14:58.9324455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:14:58.9545055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:14:58.9700359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:14:59.1305113Z skip (1.048s) 2022-03-17T19:14:59.1305352Z 2022-03-17T19:14:59.1305770Z ---------------------------------------------------------------------- 2022-03-17T19:14:59.1306081Z Ran 3 tests in 4.801s 2022-03-17T19:14:59.1306253Z 2022-03-17T19:14:59.1306343Z OK (skipped=3) 2022-03-17T19:14:59.1306472Z 2022-03-17T19:14:59.1306595Z Generating XML reports... 2022-03-17T19:14:59.1353271Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220317191454.xml 2022-03-17T19:14:59.3707036Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-03-17 19:14:59.370206] 2022-03-17T19:14:59.3708069Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:14:59.370310] 2022-03-17T19:15:00.1629919Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype 2022-03-17T19:15:00.1646069Z 2022-03-17T19:15:00.1646291Z Running tests... 2022-03-17T19:15:00.1646723Z ---------------------------------------------------------------------- 2022-03-17T19:15:01.6785695Z test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:15:01.7170893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3127 2022-03-17T19:15:01.7302284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3128 2022-03-17T19:15:01.7409228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3129 2022-03-17T19:15:01.7515307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3130 2022-03-17T19:15:02.5898013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:02.6036832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:02.6176062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:02.6217916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:02.7553777Z skip (2.590s) 2022-03-17T19:15:02.7554248Z 2022-03-17T19:15:02.7554969Z ---------------------------------------------------------------------- 2022-03-17T19:15:02.7555331Z Ran 1 test in 2.591s 2022-03-17T19:15:02.7555481Z 2022-03-17T19:15:02.7555589Z OK (skipped=1) 2022-03-17T19:15:02.7555741Z 2022-03-17T19:15:02.7555871Z Generating XML reports... 2022-03-17T19:15:02.7597851Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220317191500.xml 2022-03-17T19:15:02.9876268Z Running distributed/_shard/sharded_tensor/test_partial_tensor ... [2022-03-17 19:15:02.987111] 2022-03-17T19:15:02.9877021Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:15:02.987216] 2022-03-17T19:15:03.7794467Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_partial_tensor 2022-03-17T19:15:03.7811990Z 2022-03-17T19:15:03.7812122Z Running tests... 2022-03-17T19:15:03.7812621Z ---------------------------------------------------------------------- 2022-03-17T19:15:05.3060040Z test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:15:05.3437864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3308 2022-03-17T19:15:05.3564794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3309 2022-03-17T19:15:05.3668257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3310 2022-03-17T19:15:05.3772981Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3311 2022-03-17T19:15:06.2223830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:06.2354794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:06.2444316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:06.2460086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:06.3811535Z skip (2.600s) 2022-03-17T19:15:06.3946654Z test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3452 2022-03-17T19:15:06.4051715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3453 2022-03-17T19:15:06.4155076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3454 2022-03-17T19:15:06.4259477Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3455 2022-03-17T19:15:07.2110571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:07.2233691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:07.2774800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:07.2797520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:07.4295328Z skip (1.048s) 2022-03-17T19:15:07.4295700Z 2022-03-17T19:15:07.4296504Z ---------------------------------------------------------------------- 2022-03-17T19:15:07.4297052Z Ran 2 tests in 3.648s 2022-03-17T19:15:07.4297238Z 2022-03-17T19:15:07.4297369Z OK (skipped=2) 2022-03-17T19:15:07.4297549Z 2022-03-17T19:15:07.4297748Z Generating XML reports... 2022-03-17T19:15:07.4341562Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_partial_tensor/TEST-TestPartialTensorReshard-20220317191503.xml 2022-03-17T19:15:07.6586583Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-03-17 19:15:07.658153] 2022-03-17T19:15:07.6587368Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:15:07.658251] 2022-03-17T19:15:08.4655888Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor 2022-03-17T19:15:08.4690360Z 2022-03-17T19:15:08.4690812Z Running tests... 2022-03-17T19:15:08.4691292Z ---------------------------------------------------------------------- 2022-03-17T19:15:09.9788256Z test_empty (__main__.TestCreateTensorFromParams) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:15:09.9946960Z ok (1.526s) 2022-03-17T19:15:10.0183722Z test_local_tensor (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3633 2022-03-17T19:15:10.0287297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3634 2022-03-17T19:15:10.0392293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3635 2022-03-17T19:15:10.0496538Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3636 2022-03-17T19:15:10.8476604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:10.8744376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:10.9135403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:10.9166296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:11.0533914Z skip (1.058s) 2022-03-17T19:15:11.0658948Z test_local_tensor_error (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3777 2022-03-17T19:15:11.0763603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3778 2022-03-17T19:15:11.0869461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3779 2022-03-17T19:15:11.0973970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3780 2022-03-17T19:15:11.9525869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:11.9754800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:11.9758676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:11.9763358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:12.2008731Z skip (1.147s) 2022-03-17T19:15:12.2134154Z test_collect_local_shard (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3921 2022-03-17T19:15:12.2240165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3922 2022-03-17T19:15:12.2344724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3923 2022-03-17T19:15:12.2449686Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3924 2022-03-17T19:15:13.0793193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:13.1214728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:13.1269067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:13.1353124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:13.3487707Z skip (1.148s) 2022-03-17T19:15:13.3619407Z test_reshard_output (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4065 2022-03-17T19:15:13.3725390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4066 2022-03-17T19:15:13.3829878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4067 2022-03-17T19:15:13.3936443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4068 2022-03-17T19:15:14.2782318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:14.2919825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:14.2927637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:14.2932432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:14.4974220Z skip (1.148s) 2022-03-17T19:15:14.5106959Z test_shard_parameter (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4209 2022-03-17T19:15:14.5212548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4210 2022-03-17T19:15:14.5316968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4211 2022-03-17T19:15:14.5421875Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4212 2022-03-17T19:15:15.3501365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:15.4098425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:15.4391074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:15.4394859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:15.6458605Z skip (1.148s) 2022-03-17T19:15:15.6599981Z test_shard_parameter_errors (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4353 2022-03-17T19:15:15.6706604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4354 2022-03-17T19:15:15.6812393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4355 2022-03-17T19:15:15.6918394Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4356 2022-03-17T19:15:16.5282457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:16.5412241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:16.5665728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:16.6185566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:16.7954679Z skip (1.149s) 2022-03-17T19:15:16.8086295Z test_shard_tensor (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4497 2022-03-17T19:15:16.8191709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4498 2022-03-17T19:15:16.8296011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4499 2022-03-17T19:15:16.8400821Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4500 2022-03-17T19:15:17.6373769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:17.6597116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:17.6643095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:17.6999677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:17.8436236Z skip (1.048s) 2022-03-17T19:15:17.8569880Z test_shard_tensor_errors (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4641 2022-03-17T19:15:17.8672979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4642 2022-03-17T19:15:17.8777066Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4643 2022-03-17T19:15:17.8881339Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4644 2022-03-17T19:15:18.7652349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:18.7666573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:18.7811789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:18.7839479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:18.9917543Z skip (1.148s) 2022-03-17T19:15:19.0049446Z test_cleanup (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4785 2022-03-17T19:15:19.0156410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4786 2022-03-17T19:15:19.0263796Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4787 2022-03-17T19:15:19.0369506Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4788 2022-03-17T19:15:19.8886110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:19.8887512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:19.8916354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:19.9319123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:20.1405189Z skip (1.149s) 2022-03-17T19:15:20.1546823Z test_complete_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4929 2022-03-17T19:15:20.1650835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4930 2022-03-17T19:15:20.1755399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4931 2022-03-17T19:15:20.1860177Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4932 2022-03-17T19:15:20.9802041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:20.9920220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:20.9971778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:21.0500116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:21.1894727Z skip (1.049s) 2022-03-17T19:15:21.1913689Z test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked) 2022-03-17T19:15:21.2028755Z Test sharded_tensor.full(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5073 2022-03-17T19:15:21.2134917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5074 2022-03-17T19:15:21.2238570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5075 2022-03-17T19:15:21.2343060Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5076 2022-03-17T19:15:22.0191038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:22.0349931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:22.0487670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:22.0513254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:22.2377131Z skip (1.048s) 2022-03-17T19:15:22.2391434Z test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked) 2022-03-17T19:15:22.2506570Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5217 2022-03-17T19:15:22.2611342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5218 2022-03-17T19:15:22.2715697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5219 2022-03-17T19:15:22.2820505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5220 2022-03-17T19:15:23.1038913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:23.1637901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:23.1842118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:23.1853947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:23.3859726Z skip (1.148s) 2022-03-17T19:15:23.3875610Z test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked) 2022-03-17T19:15:23.3991269Z Test sharded_tensor.rand(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5361 2022-03-17T19:15:23.4097936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5362 2022-03-17T19:15:23.4206312Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5363 2022-03-17T19:15:23.4311028Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5364 2022-03-17T19:15:24.2271811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:24.2497929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:24.2498504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:24.3094879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:24.4346077Z skip (1.048s) 2022-03-17T19:15:24.4359478Z test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked) 2022-03-17T19:15:24.4475599Z Test sharded_tensor.zeros(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5505 2022-03-17T19:15:24.4581035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5506 2022-03-17T19:15:24.4684911Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5507 2022-03-17T19:15:24.4789562Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5508 2022-03-17T19:15:25.2729058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:25.2975635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:25.2976905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:25.3552523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:25.4823371Z skip (1.048s) 2022-03-17T19:15:25.4836221Z test_gather_even (__main__.TestShardedTensorChunked) 2022-03-17T19:15:25.4952493Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5649 2022-03-17T19:15:25.5056227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5650 2022-03-17T19:15:25.5161547Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5651 2022-03-17T19:15:25.5270900Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5652 2022-03-17T19:15:26.3421196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:26.3521321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:26.3650132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:26.4168621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:26.6308189Z skip (1.148s) 2022-03-17T19:15:26.6321636Z test_gather_uneven (__main__.TestShardedTensorChunked) 2022-03-17T19:15:26.6437902Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5793 2022-03-17T19:15:26.6543406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5794 2022-03-17T19:15:26.6648061Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5795 2022-03-17T19:15:26.6753611Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5796 2022-03-17T19:15:27.5120805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:27.5737559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:27.5823281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:27.5842085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:27.7792218Z skip (1.148s) 2022-03-17T19:15:27.7927122Z test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5937 2022-03-17T19:15:27.8034728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5938 2022-03-17T19:15:27.8140726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5939 2022-03-17T19:15:27.8247267Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5940 2022-03-17T19:15:28.6270456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:28.6793040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:28.6960493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:28.6986723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:28.8281389Z skip (1.049s) 2022-03-17T19:15:28.8410427Z test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6081 2022-03-17T19:15:28.8517175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6082 2022-03-17T19:15:28.8622361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6083 2022-03-17T19:15:28.8728736Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6084 2022-03-17T19:15:29.7120714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:29.7125827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:29.7170130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:29.7581539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:29.9765152Z skip (1.148s) 2022-03-17T19:15:29.9908690Z test_invalid_sharding (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6225 2022-03-17T19:15:30.0013454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6226 2022-03-17T19:15:30.0117574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6227 2022-03-17T19:15:30.0222928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6228 2022-03-17T19:15:30.8605315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:30.8651358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:30.8759612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:30.8767275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:31.0258005Z skip (1.049s) 2022-03-17T19:15:31.0392933Z test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6369 2022-03-17T19:15:31.0500571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6370 2022-03-17T19:15:31.0606608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6371 2022-03-17T19:15:31.0716083Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6372 2022-03-17T19:15:31.9045009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:31.9126788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:31.9307952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:31.9543255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:32.1755106Z skip (1.150s) 2022-03-17T19:15:32.1893768Z test_multiple_local_shards (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6513 2022-03-17T19:15:32.2000218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6514 2022-03-17T19:15:32.2105971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6515 2022-03-17T19:15:32.2211957Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6516 2022-03-17T19:15:33.0430789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:33.0596227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:33.0977561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:33.1004969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:33.2244621Z skip (1.049s) 2022-03-17T19:15:33.2386332Z test_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6657 2022-03-17T19:15:33.2490932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6658 2022-03-17T19:15:33.2599641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6659 2022-03-17T19:15:33.2704628Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6660 2022-03-17T19:15:34.0685637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:34.1508889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:34.1676978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:34.1683954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:34.3740486Z skip (1.149s) 2022-03-17T19:15:34.3878130Z test_partial_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6801 2022-03-17T19:15:34.3982440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6802 2022-03-17T19:15:34.4086811Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6803 2022-03-17T19:15:34.4192974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6804 2022-03-17T19:15:35.2220069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:35.2257403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:35.2323903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:35.2898991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:35.4227412Z skip (1.049s) 2022-03-17T19:15:35.4361769Z test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6945 2022-03-17T19:15:35.4466448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6946 2022-03-17T19:15:35.4570769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6947 2022-03-17T19:15:35.4677673Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6948 2022-03-17T19:15:36.3287274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:36.3484301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:36.3489662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:36.3546187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:36.5714598Z skip (1.149s) 2022-03-17T19:15:36.5853088Z test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7089 2022-03-17T19:15:36.5960305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7090 2022-03-17T19:15:36.6068158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7091 2022-03-17T19:15:36.6172181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7092 2022-03-17T19:15:37.4652121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:37.4659148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:37.5177101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:37.5186461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:37.7210000Z skip (1.149s) 2022-03-17T19:15:37.7343487Z test_sharding_columns (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7233 2022-03-17T19:15:37.7450330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7234 2022-03-17T19:15:37.7555186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7235 2022-03-17T19:15:37.7660291Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7236 2022-03-17T19:15:38.5807706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:38.5841289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:38.5842196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:38.5862268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:38.7695076Z skip (1.048s) 2022-03-17T19:15:38.7822641Z test_state_dict (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7377 2022-03-17T19:15:38.7926964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7378 2022-03-17T19:15:38.8031418Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7379 2022-03-17T19:15:38.8135972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7380 2022-03-17T19:15:39.6446335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:39.6756997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:39.6799360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:39.7026852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:39.9172557Z skip (1.148s) 2022-03-17T19:15:39.9304148Z test_state_dict_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7521 2022-03-17T19:15:39.9409213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7522 2022-03-17T19:15:39.9514200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7523 2022-03-17T19:15:39.9619420Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7524 2022-03-17T19:15:40.7742123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:40.7811925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:40.8064550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:40.8667609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:41.0655255Z skip (1.148s) 2022-03-17T19:15:41.0786547Z test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7665 2022-03-17T19:15:41.0890934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7666 2022-03-17T19:15:41.0996513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7667 2022-03-17T19:15:41.1101468Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7668 2022-03-17T19:15:41.9081512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:41.9094959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:41.9157038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:41.9576039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:42.1135978Z skip (1.048s) 2022-03-17T19:15:42.1265038Z test_custom_op (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7809 2022-03-17T19:15:42.1369950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7810 2022-03-17T19:15:42.1475572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7811 2022-03-17T19:15:42.1580453Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7812 2022-03-17T19:15:42.9557612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:42.9642850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:42.9714665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:43.0062747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:43.1616554Z skip (1.048s) 2022-03-17T19:15:43.1744050Z test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7953 2022-03-17T19:15:43.1851842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7954 2022-03-17T19:15:43.1958057Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7955 2022-03-17T19:15:43.2063030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7956 2022-03-17T19:15:44.0116786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:44.1030861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:44.1075588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:44.1076497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:44.3099306Z skip (1.148s) 2022-03-17T19:15:44.3227578Z test_custom_op_override (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8097 2022-03-17T19:15:44.3332853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8098 2022-03-17T19:15:44.3439124Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8099 2022-03-17T19:15:44.3544986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8100 2022-03-17T19:15:45.1881315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:45.1903138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:45.2186808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:45.2385484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:45.4581909Z skip (1.148s) 2022-03-17T19:15:45.4600838Z test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable) 2022-03-17T19:15:45.4718011Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8241 2022-03-17T19:15:45.4824713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8242 2022-03-17T19:15:45.4934154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8243 2022-03-17T19:15:45.5042183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8244 2022-03-17T19:15:46.3244385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:46.3535892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:46.3536426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:46.3926075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:46.6079063Z skip (1.150s) 2022-03-17T19:15:46.6096517Z test_gather_even (__main__.TestShardedTensorEnumerable) 2022-03-17T19:15:46.6215866Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8385 2022-03-17T19:15:46.6323158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8386 2022-03-17T19:15:46.6429207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8387 2022-03-17T19:15:46.6534279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8388 2022-03-17T19:15:47.4545585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:47.4726001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:47.4730443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:47.4750197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:47.6569140Z skip (1.049s) 2022-03-17T19:15:47.6586783Z test_gather_uneven (__main__.TestShardedTensorEnumerable) 2022-03-17T19:15:47.6703122Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8529 2022-03-17T19:15:47.6808044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8530 2022-03-17T19:15:47.6913094Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8531 2022-03-17T19:15:47.7020413Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8532 2022-03-17T19:15:48.5426127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:48.5693302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:48.5797069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:48.5886740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:48.8057688Z skip (1.149s) 2022-03-17T19:15:48.8207647Z test_grid_sharding (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8673 2022-03-17T19:15:48.8316836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8674 2022-03-17T19:15:48.8424696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8675 2022-03-17T19:15:48.8530902Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8676 2022-03-17T19:15:49.6959109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:49.6996859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:49.7340802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:49.7354688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:49.9568001Z skip (1.151s) 2022-03-17T19:15:49.9716032Z test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8817 2022-03-17T19:15:49.9821624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8818 2022-03-17T19:15:49.9927353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8819 2022-03-17T19:15:50.0031825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8820 2022-03-17T19:15:50.8106383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:50.8144091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:50.8586005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:50.8642860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:51.0066549Z skip (1.050s) 2022-03-17T19:15:51.0210239Z test_new_group (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8961 2022-03-17T19:15:51.0313851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8962 2022-03-17T19:15:51.0417700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8963 2022-03-17T19:15:51.0522532Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8964 2022-03-17T19:15:51.8531464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:51.8571990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:51.8681111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:51.9341512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:52.0556775Z skip (1.049s) 2022-03-17T19:15:52.0700622Z test_partial_world_size (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9105 2022-03-17T19:15:52.0803709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9106 2022-03-17T19:15:52.0908321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9107 2022-03-17T19:15:52.1012487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9108 2022-03-17T19:15:52.9021100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:52.9075854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:52.9076280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:52.9628089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:53.1047345Z skip (1.049s) 2022-03-17T19:15:53.1189961Z test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9249 2022-03-17T19:15:53.1294217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9250 2022-03-17T19:15:53.1397630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9251 2022-03-17T19:15:53.1502361Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9252 2022-03-17T19:15:53.9576394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:54.0276162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:54.0350021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:54.0405116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:54.2542018Z skip (1.149s) 2022-03-17T19:15:54.2689415Z test_uneven_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9393 2022-03-17T19:15:54.2796576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9394 2022-03-17T19:15:54.2902674Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9395 2022-03-17T19:15:54.3007949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9396 2022-03-17T19:15:55.1200854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:55.1519724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:55.1652718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:55.1699227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:55.3043249Z skip (1.050s) 2022-03-17T19:15:55.3189173Z test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9537 2022-03-17T19:15:55.3296874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9538 2022-03-17T19:15:55.3402084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9539 2022-03-17T19:15:55.3507362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9540 2022-03-17T19:15:56.1664679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:56.1799177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:56.2103617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:56.2223195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:56.3543126Z skip (1.050s) 2022-03-17T19:15:56.3684570Z test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9681 2022-03-17T19:15:56.3788433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9682 2022-03-17T19:15:56.3891734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9683 2022-03-17T19:15:56.3996126Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9684 2022-03-17T19:15:57.1842935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:57.2008758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:57.2069787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:57.2740616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:57.4030729Z skip (1.049s) 2022-03-17T19:15:57.4180006Z test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9825 2022-03-17T19:15:57.4284790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9826 2022-03-17T19:15:57.4391110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9827 2022-03-17T19:15:57.4495438Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9828 2022-03-17T19:15:58.2361865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:58.2426528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:58.2655282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:58.3238762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:58.4529450Z skip (1.050s) 2022-03-17T19:15:58.4687768Z test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9969 2022-03-17T19:15:58.4792804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9970 2022-03-17T19:15:58.4897707Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9971 2022-03-17T19:15:58.5002603Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9972 2022-03-17T19:15:59.3216984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:15:59.3323217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:15:59.3812943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:15:59.4087634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:15:59.6040451Z skip (1.151s) 2022-03-17T19:15:59.6180091Z test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10113 2022-03-17T19:15:59.6286968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10114 2022-03-17T19:15:59.6393474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10115 2022-03-17T19:15:59.6501487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10116 2022-03-17T19:16:00.4640625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:00.4646877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:00.5056185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:00.5351727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:00.7538861Z skip (1.150s) 2022-03-17T19:16:00.7669884Z test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10257 2022-03-17T19:16:00.7775707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10258 2022-03-17T19:16:00.7881075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10259 2022-03-17T19:16:00.7986177Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10260 2022-03-17T19:16:01.6981102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:01.7056478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:01.7223895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:01.7550211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:01.7660886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:01.7699212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:16:01.7699777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:01.7700198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:16:01.7701385Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:16:01.7702404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:16:01.7703100Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:16:01.7763577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:16:02.0026324Z skip (1.249s) 2022-03-17T19:16:02.0164557Z test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10413 2022-03-17T19:16:02.0269967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10414 2022-03-17T19:16:02.0375934Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10415 2022-03-17T19:16:02.0482033Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10416 2022-03-17T19:16:02.8510102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:02.8548303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:02.8852941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:02.9245457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:03.0517699Z skip (1.049s) 2022-03-17T19:16:03.0645066Z test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10557 2022-03-17T19:16:03.0749876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10558 2022-03-17T19:16:03.0855212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10559 2022-03-17T19:16:03.0960809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10560 2022-03-17T19:16:03.9051987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:03.9117508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:03.9135219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:03.9141171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:04.0995276Z skip (1.048s) 2022-03-17T19:16:04.1121032Z test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10701 2022-03-17T19:16:04.1227629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10702 2022-03-17T19:16:04.1333617Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10703 2022-03-17T19:16:04.1440671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10704 2022-03-17T19:16:05.0173274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:05.0235978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:05.0302071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:05.0517016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:05.2479425Z skip (1.148s) 2022-03-17T19:16:05.2616654Z test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10845 2022-03-17T19:16:05.2723402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10846 2022-03-17T19:16:05.2828412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10847 2022-03-17T19:16:05.2933199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10848 2022-03-17T19:16:06.0943145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:06.1077523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:06.1102655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:06.1624644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:06.2968438Z skip (1.049s) 2022-03-17T19:16:06.3097948Z test_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10989 2022-03-17T19:16:06.3202608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10990 2022-03-17T19:16:06.3308140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10991 2022-03-17T19:16:06.3413855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10992 2022-03-17T19:16:07.1519123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:07.1548487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:07.1551674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:07.1794576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:07.3449512Z skip (1.048s) 2022-03-17T19:16:07.3576177Z test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11133 2022-03-17T19:16:07.3681676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11134 2022-03-17T19:16:07.3789979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11135 2022-03-17T19:16:07.3895263Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11136 2022-03-17T19:16:08.2462988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:08.2831841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:08.2861263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:08.2897421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:08.4932894Z skip (1.148s) 2022-03-17T19:16:08.5062952Z test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11277 2022-03-17T19:16:08.5170160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11278 2022-03-17T19:16:08.5275713Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11279 2022-03-17T19:16:08.5382103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11280 2022-03-17T19:16:09.3364813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:09.3481882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:09.3793845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:09.3806622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:09.5417080Z skip (1.048s) 2022-03-17T19:16:09.5986543Z test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.057s) 2022-03-17T19:16:09.5987574Z 2022-03-17T19:16:09.5988027Z ---------------------------------------------------------------------- 2022-03-17T19:16:09.5988374Z Ran 56 tests in 61.130s 2022-03-17T19:16:09.5988541Z 2022-03-17T19:16:09.5988643Z OK (skipped=54) 2022-03-17T19:16:09.5991586Z 2022-03-17T19:16:09.5991960Z Generating XML reports... 2022-03-17T19:16:09.6032203Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20220317191508.xml 2022-03-17T19:16:09.6036499Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20220317191508.xml 2022-03-17T19:16:09.6042111Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20220317191508.xml 2022-03-17T19:16:09.6047884Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20220317191508.xml 2022-03-17T19:16:09.6053718Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20220317191508.xml 2022-03-17T19:16:09.6059479Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20220317191508.xml 2022-03-17T19:16:09.6092895Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20220317191508.xml 2022-03-17T19:16:09.6099949Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20220317191508.xml 2022-03-17T19:16:09.6117373Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20220317191508.xml 2022-03-17T19:16:09.6134129Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20220317191508.xml 2022-03-17T19:16:09.6139747Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20220317191508.xml 2022-03-17T19:16:09.8455465Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-03-17 19:16:09.845048] 2022-03-17T19:16:09.8456420Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:16:09.845151] 2022-03-17T19:16:10.6456311Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard 2022-03-17T19:16:10.6473743Z 2022-03-17T19:16:10.6474180Z Running tests... 2022-03-17T19:16:10.6474789Z ---------------------------------------------------------------------- 2022-03-17T19:16:12.1785096Z test_sharded_tensor_reshard (__main__.TestReshard) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:16:12.2164682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11458 2022-03-17T19:16:12.2293287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11459 2022-03-17T19:16:12.2397660Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11460 2022-03-17T19:16:12.2503243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11461 2022-03-17T19:16:13.0983281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:13.1034548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:13.1102637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:13.1199933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:13.2546101Z skip (2.607s) 2022-03-17T19:16:13.2677098Z test_sharded_tensor_reshard_errors (__main__.TestReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11602 2022-03-17T19:16:13.2781380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11603 2022-03-17T19:16:13.2885893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11604 2022-03-17T19:16:13.2990576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11605 2022-03-17T19:16:14.0950940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:14.0967724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:14.0968254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:14.0990666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:14.3029785Z skip (1.048s) 2022-03-17T19:16:14.3030049Z 2022-03-17T19:16:14.3030481Z ---------------------------------------------------------------------- 2022-03-17T19:16:14.3030808Z Ran 2 tests in 3.655s 2022-03-17T19:16:14.3030979Z 2022-03-17T19:16:14.3031104Z OK (skipped=2) 2022-03-17T19:16:14.3031232Z 2022-03-17T19:16:14.3031282Z Generating XML reports... 2022-03-17T19:16:14.3076682Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220317191610.xml 2022-03-17T19:16:14.5403690Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-03-17 19:16:14.539867] 2022-03-17T19:16:14.5404383Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:16:14.539969] 2022-03-17T19:16:15.3395936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc31sr4ry 2022-03-17T19:16:15.3397001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc31sr4ry/_remote_module_non_sriptable.py 2022-03-17T19:16:15.3502982Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec 2022-03-17T19:16:15.3524210Z 2022-03-17T19:16:15.3524717Z Running tests... 2022-03-17T19:16:15.3525180Z ---------------------------------------------------------------------- 2022-03-17T19:16:16.8833916Z test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:16:16.9215808Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11783 2022-03-17T19:16:16.9321209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11784 2022-03-17T19:16:16.9426944Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11785 2022-03-17T19:16:16.9532319Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11786 2022-03-17T19:16:17.7403741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zkbixee 2022-03-17T19:16:17.7404869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zkbixee/_remote_module_non_sriptable.py 2022-03-17T19:16:17.7483367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:17.7594687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdl2sv78g 2022-03-17T19:16:17.7596512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdl2sv78g/_remote_module_non_sriptable.py 2022-03-17T19:16:17.7678025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:17.8201674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9bpg80p 2022-03-17T19:16:17.8202788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9bpg80p/_remote_module_non_sriptable.py 2022-03-17T19:16:17.8283302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:17.8322842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhcicb9t 2022-03-17T19:16:17.8324850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhcicb9t/_remote_module_non_sriptable.py 2022-03-17T19:16:17.8408724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:17.9571216Z ok (2.604s) 2022-03-17T19:16:17.9598338Z test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec) 2022-03-17T19:16:17.9715305Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11927 2022-03-17T19:16:17.9821621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11928 2022-03-17T19:16:17.9926658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11929 2022-03-17T19:16:18.0033448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11930 2022-03-17T19:16:18.7798075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmeo3p0fy 2022-03-17T19:16:18.7798976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmeo3p0fy/_remote_module_non_sriptable.py 2022-03-17T19:16:18.7845447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp83k0i9u 2022-03-17T19:16:18.7848519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp83k0i9u/_remote_module_non_sriptable.py 2022-03-17T19:16:18.7877685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:18.7928993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:18.8046059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrmjbh8v 2022-03-17T19:16:18.8047559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrmjbh8v/_remote_module_non_sriptable.py 2022-03-17T19:16:18.8129727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:18.8204361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nn0k_pi 2022-03-17T19:16:18.8205461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nn0k_pi/_remote_module_non_sriptable.py 2022-03-17T19:16:18.8289400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:19.0068848Z skip (1.050s) 2022-03-17T19:16:19.0080939Z test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec) 2022-03-17T19:16:19.0197080Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12071 2022-03-17T19:16:19.0302814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12072 2022-03-17T19:16:19.0408395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12073 2022-03-17T19:16:19.0514133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12074 2022-03-17T19:16:19.8226214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalz060v5 2022-03-17T19:16:19.8227046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalz060v5/_remote_module_non_sriptable.py 2022-03-17T19:16:19.8300187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawsl39md 2022-03-17T19:16:19.8303026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawsl39md/_remote_module_non_sriptable.py 2022-03-17T19:16:19.8305630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:16:19.8382150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:19.8888354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvgg087j 2022-03-17T19:16:19.8889260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvgg087j/_remote_module_non_sriptable.py 2022-03-17T19:16:19.8972818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:16:19.8990195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0r_0pxe 2022-03-17T19:16:19.8992950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0r_0pxe/_remote_module_non_sriptable.py 2022-03-17T19:16:19.9075471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:20.0549213Z skip (1.048s) 2022-03-17T19:16:20.0700932Z test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.015s) 2022-03-17T19:16:20.0788480Z test_device_placement (__main__.TestShardingSpec) ... ok (0.009s) 2022-03-17T19:16:20.0883053Z test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.009s) 2022-03-17T19:16:20.0905682Z test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s) 2022-03-17T19:16:20.0918366Z test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s) 2022-03-17T19:16:20.0931485Z test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s) 2022-03-17T19:16:20.1037684Z test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.011s) 2022-03-17T19:16:20.1038064Z 2022-03-17T19:16:20.1038451Z ---------------------------------------------------------------------- 2022-03-17T19:16:20.1038809Z Ran 10 tests in 4.751s 2022-03-17T19:16:20.1038989Z 2022-03-17T19:16:20.1039548Z OK (skipped=2) 2022-03-17T19:16:20.1039746Z 2022-03-17T19:16:20.1039877Z Generating XML reports... 2022-03-17T19:16:20.1084575Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220317191615.xml 2022-03-17T19:16:20.1095394Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220317191615.xml 2022-03-17T19:16:20.3444336Z Running distributed/algorithms/test_join ... [2022-03-17 19:16:20.343934] 2022-03-17T19:16:20.3445041Z Executing ['/opt/conda/bin/python', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:16:20.344036] 2022-03-17T19:16:21.1299019Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join 2022-03-17T19:16:21.1316062Z 2022-03-17T19:16:21.1316528Z Running tests... 2022-03-17T19:16:21.1317020Z ---------------------------------------------------------------------- 2022-03-17T19:16:21.1328065Z test_join_kwargs (__main__.TestJoin) 2022-03-17T19:16:22.6566081Z Tests passing keyword arguments to the context manager. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:16:22.6948977Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12252 2022-03-17T19:16:22.7053662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12253 2022-03-17T19:16:23.4949328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:23.4951963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:23.4996913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:23.5000926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:23.5002035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:23.5054375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:26.2132909Z ok (5.081s) 2022-03-17T19:16:26.2143509Z test_multiple_joinable_disable (__main__.TestJoin) 2022-03-17T19:16:26.2266670Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12336 2022-03-17T19:16:26.2372994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12337 2022-03-17T19:16:27.0170640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:27.0173361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:27.0285334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:27.0289005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:27.0289798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:27.0377519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:29.7450783Z ok (3.532s) 2022-03-17T19:16:29.7461957Z test_multiple_joinables (__main__.TestJoin) 2022-03-17T19:16:29.7583299Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12420 2022-03-17T19:16:29.7688966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12421 2022-03-17T19:16:30.5503103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:30.5505641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:30.5611512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:30.5615185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:30.5616092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:30.5710335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:33.2766238Z ok (3.531s) 2022-03-17T19:16:33.2774590Z test_multiple_joinables_throw (__main__.TestJoin) 2022-03-17T19:16:33.2892251Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12504 2022-03-17T19:16:33.2996846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12505 2022-03-17T19:16:34.0781004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:34.0783285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:34.1256271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:34.1260217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:34.1261240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:34.1291222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:36.8073359Z ok (3.531s) 2022-03-17T19:16:36.8083717Z test_single_joinable (__main__.TestJoin) 2022-03-17T19:16:36.8200679Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12588 2022-03-17T19:16:36.8304185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12589 2022-03-17T19:16:37.6289108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:37.6291858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:37.6514556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:37.6518129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:37.6518953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:37.6597717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:40.3380604Z ok (3.531s) 2022-03-17T19:16:40.3389406Z test_single_joinable_disable (__main__.TestJoin) 2022-03-17T19:16:40.3505133Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12672 2022-03-17T19:16:40.3609145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12673 2022-03-17T19:16:41.1454095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:41.1456572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:41.1947443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:41.1951600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:41.1952428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:41.1964600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:43.8688001Z ok (3.531s) 2022-03-17T19:16:43.8698427Z test_single_joinable_main_hooks (__main__.TestJoin) 2022-03-17T19:16:43.8821596Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12756 2022-03-17T19:16:43.8929548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12757 2022-03-17T19:16:44.6777371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:44.6780444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:44.7337825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:44.7341516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:44.7342545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:44.7389384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:47.4006543Z ok (3.532s) 2022-03-17T19:16:47.4014438Z test_single_joinable_post_hooks (__main__.TestJoin) 2022-03-17T19:16:47.4133955Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12840 2022-03-17T19:16:47.4240672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12841 2022-03-17T19:16:48.2442865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:48.2443344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:48.2444059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:48.2446525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:48.2447302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:48.2546991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:50.9316722Z ok (3.531s) 2022-03-17T19:16:50.9324600Z test_single_joinable_throw (__main__.TestJoin) 2022-03-17T19:16:50.9442192Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12924 2022-03-17T19:16:50.9547807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12925 2022-03-17T19:16:51.7497879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:16:51.7500619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:16:51.7882621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:16:51.7887126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:16:51.7888144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:51.7906641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:16:54.4624774Z ok (3.531s) 2022-03-17T19:16:54.4625145Z 2022-03-17T19:16:54.4627913Z ---------------------------------------------------------------------- 2022-03-17T19:16:54.4628300Z Ran 9 tests in 33.331s 2022-03-17T19:16:54.4628481Z 2022-03-17T19:16:54.4628578Z OK 2022-03-17T19:16:54.4628696Z 2022-03-17T19:16:54.4628829Z Generating XML reports... 2022-03-17T19:16:54.4691480Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220317191621.xml 2022-03-17T19:16:54.6954101Z Running distributed/elastic/events/lib_test ... [2022-03-17 19:16:54.694929] 2022-03-17T19:16:54.6954733Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-03-17 19:16:54.695031] 2022-03-17T19:16:55.2872894Z ============================= test session starts ============================== 2022-03-17T19:16:55.2873566Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:16:55.2932286Z cachedir: .pytest_cache 2022-03-17T19:16:55.2933203Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:16:55.2933637Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:16:55.2934222Z plugins: hypothesis-4.53.2 2022-03-17T19:16:56.0122487Z collecting ...  2022-03-17T19:16:56.0135718Z collecting 3 items  2022-03-17T19:16:56.0136194Z collected 8 items  2022-03-17T19:16:56.0141493Z 2022-03-17T19:16:56.0159097Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%] 2022-03-17T19:16:56.0174740Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%] 2022-03-17T19:16:56.0195129Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%] 2022-03-17T19:16:56.0827737Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%] 2022-03-17T19:16:56.0846003Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%] 2022-03-17T19:16:56.0860066Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%] 2022-03-17T19:16:56.0875389Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%] 2022-03-17T19:16:56.0897357Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%] 2022-03-17T19:16:56.0899582Z 2022-03-17T19:16:56.0900068Z ============================== 8 passed in 0.80s =============================== 2022-03-17T19:16:56.1983602Z Running distributed/elastic/metrics/api_test ... [2022-03-17 19:16:56.197890] 2022-03-17T19:16:56.1984311Z Executing ['/opt/conda/bin/python', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:16:56.197991] 2022-03-17T19:16:56.9814310Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test 2022-03-17T19:16:56.9832060Z 2022-03-17T19:16:56.9832483Z Running tests... 2022-03-17T19:16:56.9832968Z ---------------------------------------------------------------------- 2022-03-17T19:16:58.5127516Z test_get_metric_name (__main__.MetricsApiTest) ... ok (1.529s) 2022-03-17T19:16:58.5141289Z test_inheritance (__main__.MetricsApiTest) ... ok (0.001s) 2022-03-17T19:16:58.5162315Z test_profile (__main__.MetricsApiTest) ... ok (0.002s) 2022-03-17T19:16:58.5162553Z 2022-03-17T19:16:58.5162949Z ---------------------------------------------------------------------- 2022-03-17T19:16:58.5163272Z Ran 3 tests in 1.533s 2022-03-17T19:16:58.5163437Z 2022-03-17T19:16:58.5163538Z OK 2022-03-17T19:16:58.5163654Z 2022-03-17T19:16:58.5163780Z Generating XML reports... 2022-03-17T19:16:58.5199505Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220317191656.xml 2022-03-17T19:16:58.7246442Z Running distributed/elastic/multiprocessing/api_test ... [2022-03-17 19:16:58.724176] 2022-03-17T19:16:58.7247199Z Executing ['/opt/conda/bin/python', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:16:58.724276] 2022-03-17T19:16:59.5911916Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test 2022-03-17T19:16:59.5933505Z 2022-03-17T19:16:59.5933684Z Running tests... 2022-03-17T19:16:59.5934138Z ---------------------------------------------------------------------- 2022-03-17T19:17:01.1320422Z test_get_failures (__main__.RunProcResultsTest) ... ok (1.538s) 2022-03-17T19:17:01.1330396Z test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s) 2022-03-17T19:17:01.1349749Z test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s) 2022-03-17T19:17:01.1849500Z test_binary (__main__.StartProcessesListTest) ... hello stdout from 0 2022-03-17T19:17:01.1849859Z hello stderr from 0 2022-03-17T19:17:01.2037771Z hello stdout from 1 2022-03-17T19:17:01.2038287Z hello stderr from 1 2022-03-17T19:17:01.2790250Z ok (0.144s) 2022-03-17T19:17:01.3470350Z test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1 2022-03-17T19:17:01.3470696Z bar stderr from 1 2022-03-17T19:17:01.4212089Z failed (exitcode: 138) local_rank: 0 (pid: 13117) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-03-17T19:17:01.4225062Z ok (0.143s) 2022-03-17T19:17:01.4439909Z test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.021s) 2022-03-17T19:17:01.4922564Z test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last): 2022-03-17T19:17:01.4923058Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-03-17T19:17:01.4923613Z raise RuntimeError(f"raised from {rank}") 2022-03-17T19:17:01.4923964Z RuntimeError: raised from 0 2022-03-17T19:17:01.5107095Z bar from 1 2022-03-17T19:17:01.5851123Z failed (exitcode: 1) local_rank: 0 (pid: 13120) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-03-17T19:17:01.5862276Z ok (0.142s) 2022-03-17T19:17:01.6554251Z test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1 2022-03-17T19:17:01.7303252Z [trainer1]:world stderr from 1 2022-03-17T19:17:01.7303586Z [trainer0]:hello stdout from 0 2022-03-17T19:17:02.7330308Z ok (1.146s) 2022-03-17T19:17:03.6562188Z test_function (__main__.StartProcessesListTest) ... hello stdout from 0 2022-03-17T19:17:03.6562550Z hello stderr from 0 2022-03-17T19:17:03.6713465Z hello stdout from 1 2022-03-17T19:17:03.6713729Z hello stderr from 1 2022-03-17T19:17:03.7862939Z Closing process 13127 via signal SIGTERM 2022-03-17T19:17:03.7881524Z ok (1.055s) 2022-03-17T19:17:05.1549121Z test_function_large_ret_val (__main__.StartProcessesListTest) ... Closing process 13197 via signal SIGTERM 2022-03-17T19:17:05.1549584Z Closing process 13198 via signal SIGTERM 2022-03-17T19:17:05.1549873Z Closing process 13199 via signal SIGTERM 2022-03-17T19:17:05.1849894Z ok (1.396s) 2022-03-17T19:17:05.1873130Z test_function_raise (__main__.StartProcessesListTest) 2022-03-17T19:17:06.2213344Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 13337) of fn: echo2 (start_method: spawn) 2022-03-17T19:17:06.2213857Z Traceback (most recent call last): 2022-03-17T19:17:06.2214515Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-03-17T19:17:06.2214958Z self._pc.join(-1) 2022-03-17T19:17:06.2215413Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-03-17T19:17:06.2215880Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-03-17T19:17:06.2216340Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-03-17T19:17:06.2216621Z 2022-03-17T19:17:06.2216837Z -- Process 0 terminated with the following error: 2022-03-17T19:17:06.2217173Z Traceback (most recent call last): 2022-03-17T19:17:06.2217672Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-03-17T19:17:06.2218025Z fn(i, *args) 2022-03-17T19:17:06.2218525Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap 2022-03-17T19:17:06.2219201Z ret = record(fn)(*args_) 2022-03-17T19:17:06.2219741Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper 2022-03-17T19:17:06.2220146Z return f(*args, **kwargs) 2022-03-17T19:17:06.2221120Z File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-03-17T19:17:06.2221508Z raise RuntimeError(msg) 2022-03-17T19:17:06.2221756Z RuntimeError: hello 2022-03-17T19:17:06.2221921Z 2022-03-17T19:17:06.2228996Z ok (1.038s) 2022-03-17T19:17:06.2255669Z test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s) 2022-03-17T19:17:06.2273295Z test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.002s) 2022-03-17T19:17:06.2406556Z test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 13407 via signal SIGTERM 2022-03-17T19:17:06.2497123Z ok (0.022s) 2022-03-17T19:17:06.2552340Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-03-17T19:17:06.2552832Z Traceback (most recent call last): 2022-03-17T19:17:06.2553353Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-03-17T19:17:06.2553784Z self._pc.join(-1) 2022-03-17T19:17:06.2554109Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-03-17T19:17:06.2554444Z return _mock_self._mock_call(*args, **kwargs) 2022-03-17T19:17:06.2554802Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-03-17T19:17:06.2555099Z raise effect 2022-03-17T19:17:06.2555467Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-03-17T19:17:06.2559729Z ok (0.006s) 2022-03-17T19:17:08.2847126Z test_pcontext_wait (__main__.StartProcessesListTest) ... ok (2.028s) 2022-03-17T19:17:08.3068280Z test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 13443 closing signal SIGTERM 2022-03-17T19:17:08.3084990Z ok (0.024s) 2022-03-17T19:17:08.3109058Z test_to_map (__main__.StartProcessesListTest) ... ok (0.002s) 2022-03-17T19:17:08.3117189Z test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s) 2022-03-17T19:17:09.2275974Z test_void_function (__main__.StartProcessesListTest) ... world 2022-03-17T19:17:09.2709946Z hello 2022-03-17T19:17:09.4451875Z Closing process 13445 via signal SIGTERM 2022-03-17T19:17:09.4463572Z ok (1.134s) 2022-03-17T19:17:09.4489854Z test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s) 2022-03-17T19:17:09.5169585Z test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1 2022-03-17T19:17:09.5169930Z bar stderr from 1 2022-03-17T19:17:09.5912328Z failed (exitcode: 138) local_rank: 0 (pid: 13514) of binary: distributed/elastic/multiprocessing/bin/echo1.py 2022-03-17T19:17:09.5924294Z ok (0.143s) 2022-03-17T19:17:09.6141609Z test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.021s) 2022-03-17T19:17:09.6618609Z test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last): 2022-03-17T19:17:09.6619069Z File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in 2022-03-17T19:17:09.6619449Z raise RuntimeError(f"raised from {rank}") 2022-03-17T19:17:09.6619735Z RuntimeError: raised from 0 2022-03-17T19:17:09.6807944Z bar from 1 2022-03-17T19:17:09.7552128Z failed (exitcode: 1) local_rank: 0 (pid: 13517) of binary: distributed/elastic/multiprocessing/bin/echo2.py 2022-03-17T19:17:09.7562807Z ok (0.142s) 2022-03-17T19:17:11.2271305Z test_function_large_ret_val (__main__.StartProcessesTest) ... Closing process 13519 via signal SIGTERM 2022-03-17T19:17:11.2271755Z Closing process 13520 via signal SIGTERM 2022-03-17T19:17:11.2272050Z Closing process 13521 via signal SIGTERM 2022-03-17T19:17:11.2558124Z ok (1.499s) 2022-03-17T19:17:11.2577899Z test_function_raise (__main__.StartProcessesTest) 2022-03-17T19:17:12.3916327Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 13659) of fn: echo2 (start_method: spawn) 2022-03-17T19:17:12.3916851Z Traceback (most recent call last): 2022-03-17T19:17:12.3917511Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-03-17T19:17:12.3917918Z self._pc.join(-1) 2022-03-17T19:17:12.3918386Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join 2022-03-17T19:17:12.3918829Z raise ProcessRaisedException(msg, error_index, failed_process.pid) 2022-03-17T19:17:12.3919561Z torch.multiprocessing.spawn.ProcessRaisedException: 2022-03-17T19:17:12.3919857Z 2022-03-17T19:17:12.3920102Z -- Process 0 terminated with the following error: 2022-03-17T19:17:12.3920407Z Traceback (most recent call last): 2022-03-17T19:17:12.3920920Z File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap 2022-03-17T19:17:12.3921263Z fn(i, *args) 2022-03-17T19:17:12.3921732Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap 2022-03-17T19:17:12.3922128Z ret = record(fn)(*args_) 2022-03-17T19:17:12.3922661Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper 2022-03-17T19:17:12.3923068Z return f(*args, **kwargs) 2022-03-17T19:17:12.3923434Z File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2 2022-03-17T19:17:12.3923814Z raise RuntimeError(msg) 2022-03-17T19:17:12.3924084Z RuntimeError: hello 2022-03-17T19:17:12.3924230Z 2022-03-17T19:17:12.3934137Z ok (1.137s) 2022-03-17T19:17:12.3962466Z test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.003s) 2022-03-17T19:17:12.3978698Z test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.002s) 2022-03-17T19:17:12.4115481Z test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 13729 via signal SIGTERM 2022-03-17T19:17:12.4205913Z ok (0.023s) 2022-03-17T19:17:12.4258403Z test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn) 2022-03-17T19:17:12.4258891Z Traceback (most recent call last): 2022-03-17T19:17:12.4259411Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll 2022-03-17T19:17:12.4259840Z self._pc.join(-1) 2022-03-17T19:17:12.4260156Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__ 2022-03-17T19:17:12.4260758Z return _mock_self._mock_call(*args, **kwargs) 2022-03-17T19:17:12.4261117Z File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call 2022-03-17T19:17:12.4261411Z raise effect 2022-03-17T19:17:12.4261788Z torch.multiprocessing.spawn.ProcessRaisedException: test msg 2022-03-17T19:17:12.4265933Z ok (0.006s) 2022-03-17T19:17:14.4484779Z test_pcontext_wait (__main__.StartProcessesTest) ... ok (2.022s) 2022-03-17T19:17:14.4706897Z test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 13765 closing signal SIGTERM 2022-03-17T19:17:14.4723589Z ok (0.024s) 2022-03-17T19:17:14.4746413Z test_to_map (__main__.StartProcessesTest) ... ok (0.002s) 2022-03-17T19:17:14.4754420Z test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s) 2022-03-17T19:17:15.3738256Z test_void_function (__main__.StartProcessesTest) ... world 2022-03-17T19:17:15.3844401Z hello 2022-03-17T19:17:15.5053498Z Closing process 13767 via signal SIGTERM 2022-03-17T19:17:15.5068380Z ok (1.031s) 2022-03-17T19:17:15.5087478Z test_from_str_bad_input (__main__.StdTest) ... ok (0.002s) 2022-03-17T19:17:15.5099169Z test_from_value (__main__.StdTest) ... ok (0.001s) 2022-03-17T19:17:15.5109824Z test_from_value_map (__main__.StdTest) ... ok (0.001s) 2022-03-17T19:17:15.5110113Z 2022-03-17T19:17:15.5110480Z ---------------------------------------------------------------------- 2022-03-17T19:17:15.5111204Z Ran 38 tests in 15.918s 2022-03-17T19:17:15.5111575Z 2022-03-17T19:17:15.5111731Z OK 2022-03-17T19:17:15.5111927Z 2022-03-17T19:17:15.5112049Z Generating XML reports... 2022-03-17T19:17:15.5167655Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220317191659.xml 2022-03-17T19:17:15.5194650Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220317191659.xml 2022-03-17T19:17:15.5216114Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220317191659.xml 2022-03-17T19:17:15.5221283Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220317191659.xml 2022-03-17T19:17:15.7532399Z Running distributed/elastic/timer/api_test ... [2022-03-17 19:17:15.752740] 2022-03-17T19:17:15.7533064Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:15.752841] 2022-03-17T19:17:16.5410308Z Running distributed/elastic/timer/local_timer_example ... [2022-03-17 19:17:16.540490] 2022-03-17T19:17:16.5411006Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:16.540590] 2022-03-17T19:17:17.3229398Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example 2022-03-17T19:17:17.3245344Z 2022-03-17T19:17:17.3245673Z Running tests... 2022-03-17T19:17:17.3246104Z ---------------------------------------------------------------------- 2022-03-17T19:17:18.8584233Z test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-03-17 19:17:18,857 driver: init 2022-03-17T19:17:18.8873043Z [INFO] 2022-03-17 19:17:18,886 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-03-17T19:17:18.8874608Z [INFO] 2022-03-17 19:17:18,887 api: Starting watchdog thread... 2022-03-17T19:17:19.8909265Z [INFO] 2022-03-17 19:17:19,890 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:19.9269622Z [INFO] 2022-03-17 19:17:19,926 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:19.9296774Z [INFO] 2022-03-17 19:17:19,929 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:19.9316649Z [INFO] 2022-03-17 19:17:19,931 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:19.9628049Z [INFO] 2022-03-17 19:17:19,962 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:19.9817816Z [INFO] 2022-03-17 19:17:19,981 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:20.0198539Z [INFO] 2022-03-17 19:17:20,019 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:20.0218456Z [INFO] 2022-03-17 19:17:20,021 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:20.9507118Z [INFO] 2022-03-17 19:17:20,949 api: Reaping worker_id=[13910]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:20.9507976Z [INFO] 2022-03-17 19:17:20,950 api: Successfully reaped worker=[13910] 2022-03-17T19:17:20.9914885Z [INFO] 2022-03-17 19:17:20,991 api: Reaping worker_id=[13912]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:20.9917657Z [INFO] 2022-03-17 19:17:20,991 api: Successfully reaped worker=[13912] 2022-03-17T19:17:21.0223688Z [INFO] 2022-03-17 19:17:21,021 api: Reaping worker_id=[13908]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:21.0225818Z [INFO] 2022-03-17 19:17:21,022 api: Successfully reaped worker=[13908] 2022-03-17T19:17:21.0838814Z [INFO] 2022-03-17 19:17:21,083 api: Reaping worker_id=[13914]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:21.0840365Z [INFO] 2022-03-17 19:17:21,083 api: Successfully reaped worker=[13914] 2022-03-17T19:17:21.0904393Z [INFO] 2022-03-17 19:17:21,090 api: Stopping LocalTimerServer 2022-03-17T19:17:21.0904842Z [INFO] 2022-03-17 19:17:21,090 api: Stopping watchdog thread... 2022-03-17T19:17:21.0951800Z ok (3.770s) 2022-03-17T19:17:21.0973872Z test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-03-17 19:17:21,097 api: Starting LocalTimerServer... max_interval=0.01, daemon=True 2022-03-17T19:17:21.0974427Z [INFO] 2022-03-17 19:17:21,097 api: Starting watchdog thread... 2022-03-17T19:17:22.1327003Z [INFO] 2022-03-17 19:17:22,132 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:22.1611568Z [INFO] 2022-03-17 19:17:22,160 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:22.1751467Z [INFO] 2022-03-17 19:17:22,174 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:22.1767780Z [INFO] 2022-03-17 19:17:22,176 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:22.1817027Z [INFO] 2022-03-17 19:17:22,181 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:22.1979224Z [INFO] 2022-03-17 19:17:22,197 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:22.2194627Z [INFO] 2022-03-17 19:17:22,219 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:22.2419792Z [INFO] 2022-03-17 19:17:22,241 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:23.9292996Z [INFO] 2022-03-17 19:17:23,928 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:23.9326774Z [INFO] 2022-03-17 19:17:23,932 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:23.9544906Z [INFO] 2022-03-17 19:17:23,953 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:23.9545939Z [INFO] 2022-03-17 19:17:23,953 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:23.9663329Z [INFO] 2022-03-17 19:17:23,965 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:23.9684454Z [INFO] 2022-03-17 19:17:23,967 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:23.9762204Z [INFO] 2022-03-17 19:17:23,975 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:23.9905289Z [INFO] 2022-03-17 19:17:23,990 api: Timer client configured to: LocalTimerClient 2022-03-17T19:17:24.9879163Z [INFO] 2022-03-17 19:17:24,987 api: Reaping worker_id=[14487]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:24.9885998Z [INFO] 2022-03-17 19:17:24,987 api: Successfully reaped worker=[14487] 2022-03-17T19:17:24.9984455Z [INFO] 2022-03-17 19:17:24,997 api: Reaping worker_id=[14490]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:24.9986939Z [INFO] 2022-03-17 19:17:24,998 api: Successfully reaped worker=[14490] 2022-03-17T19:17:25.0190980Z [INFO] 2022-03-17 19:17:25,018 api: Reaping worker_id=[14488]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:25.0191553Z [INFO] 2022-03-17 19:17:25,018 local_timer: Process with pid=14488 does not exist. Skipping 2022-03-17T19:17:25.0192012Z [INFO] 2022-03-17 19:17:25,018 api: Successfully reaped worker=[14488] 2022-03-17T19:17:25.0192578Z [INFO] 2022-03-17 19:17:25,018 api: Reaping worker_id=[14486]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:25.0193099Z [INFO] 2022-03-17 19:17:25,018 local_timer: Process with pid=14486 does not exist. Skipping 2022-03-17T19:17:25.0193571Z [INFO] 2022-03-17 19:17:25,019 api: Successfully reaped worker=[14486] 2022-03-17T19:17:25.0295839Z [INFO] 2022-03-17 19:17:25,029 api: Reaping worker_id=[14489]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:25.0296388Z [INFO] 2022-03-17 19:17:25,029 local_timer: Process with pid=14489 does not exist. Skipping 2022-03-17T19:17:25.0296863Z [INFO] 2022-03-17 19:17:25,029 api: Successfully reaped worker=[14489] 2022-03-17T19:17:25.0358992Z [INFO] 2022-03-17 19:17:25,035 api: Stopping LocalTimerServer 2022-03-17T19:17:25.0359727Z [INFO] 2022-03-17 19:17:25,035 api: Stopping watchdog thread... 2022-03-17T19:17:25.0398998Z [INFO] 2022-03-17 19:17:25,039 api: Reaping worker_id=[14485]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:25.0399569Z [INFO] 2022-03-17 19:17:25,039 local_timer: Process with pid=14485 does not exist. Skipping 2022-03-17T19:17:25.0400028Z [INFO] 2022-03-17 19:17:25,039 api: Successfully reaped worker=[14485] 2022-03-17T19:17:25.0400594Z [INFO] 2022-03-17 19:17:25,039 api: Reaping worker_id=[14491]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112'] 2022-03-17T19:17:25.0401242Z [INFO] 2022-03-17 19:17:25,039 local_timer: Process with pid=14491 does not exist. Skipping 2022-03-17T19:17:25.0402075Z [INFO] 2022-03-17 19:17:25,039 api: Successfully reaped worker=[14491] 2022-03-17T19:17:25.0407638Z ok (3.945s) 2022-03-17T19:17:25.0410257Z 2022-03-17T19:17:25.0410873Z ---------------------------------------------------------------------- 2022-03-17T19:17:25.0411256Z Ran 2 tests in 7.716s 2022-03-17T19:17:25.0411430Z 2022-03-17T19:17:25.0411511Z OK 2022-03-17T19:17:25.0411604Z 2022-03-17T19:17:25.0411710Z Generating XML reports... 2022-03-17T19:17:25.0454659Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220317191717.xml 2022-03-17T19:17:25.2733094Z Running distributed/elastic/timer/local_timer_test ... [2022-03-17 19:17:25.272776] 2022-03-17T19:17:25.2733825Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:25.272876] 2022-03-17T19:17:26.0905586Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test 2022-03-17T19:17:26.0923983Z 2022-03-17T19:17:26.0924426Z Running tests... 2022-03-17T19:17:26.0924907Z ---------------------------------------------------------------------- 2022-03-17T19:17:26.0934877Z test_acquire_release (__main__.LocalTimerServerTest) 2022-03-17T19:17:27.6359707Z tests that: ... ok (1.543s) 2022-03-17T19:17:27.6367304Z test_expired_timers (__main__.LocalTimerServerTest) 2022-03-17T19:17:27.6384903Z tests that a single expired timer on a process should terminate ... ok (0.002s) 2022-03-17T19:17:27.6398727Z test_valid_timers (__main__.LocalTimerServerTest) 2022-03-17T19:17:27.6415805Z tests that valid timers are processed correctly and the process is left alone ... ok (0.003s) 2022-03-17T19:17:27.6424839Z test_watchdog_call_count (__main__.LocalTimerServerTest) 2022-03-17T19:17:27.7454301Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.104s) 2022-03-17T19:17:27.7457102Z test_watchdog_empty_queue (__main__.LocalTimerServerTest) 2022-03-17T19:17:27.7564827Z checks that the watchdog can run on an empty queue ... ok (0.011s) 2022-03-17T19:17:27.7731958Z test_client_interaction (__main__.LocalTimerTest) ... ok (0.016s) 2022-03-17T19:17:27.7855311Z test_exception_propagation (__main__.LocalTimerTest) ... ok (0.012s) 2022-03-17T19:17:27.7867362Z test_get_timer_recursive (__main__.LocalTimerTest) 2022-03-17T19:17:29.0424593Z If a function acquires a countdown timer with default scope, ... ok (1.256s) 2022-03-17T19:17:29.1467984Z test_happy_path (__main__.LocalTimerTest) ... ok (0.104s) 2022-03-17T19:17:29.1582801Z test_no_client (__main__.LocalTimerTest) ... ok (0.011s) 2022-03-17T19:17:29.3064534Z test_timer (__main__.LocalTimerTest) ... ok (0.148s) 2022-03-17T19:17:29.3298783Z test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.023s) 2022-03-17T19:17:29.3310301Z test_get_less_than_size (__main__.MultiprocessingRequestQueueTest) 2022-03-17T19:17:29.8439078Z Tests slow producer. ... ok (0.514s) 2022-03-17T19:17:29.8457352Z test_get_size (__main__.MultiprocessingRequestQueueTest) 2022-03-17T19:17:30.7613643Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.917s) 2022-03-17T19:17:30.7617039Z 2022-03-17T19:17:30.7617617Z ---------------------------------------------------------------------- 2022-03-17T19:17:30.7617967Z Ran 14 tests in 4.669s 2022-03-17T19:17:30.7618141Z 2022-03-17T19:17:30.7618212Z OK 2022-03-17T19:17:30.7618344Z 2022-03-17T19:17:30.7618433Z Generating XML reports... 2022-03-17T19:17:30.7707192Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220317191726.xml 2022-03-17T19:17:30.7716199Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220317191726.xml 2022-03-17T19:17:30.7723000Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220317191726.xml 2022-03-17T19:17:31.1042667Z Running distributed/elastic/utils/distributed_test ... [2022-03-17 19:17:31.103710] 2022-03-17T19:17:31.1043427Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:31.103813] 2022-03-17T19:17:31.9186954Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test 2022-03-17T19:17:31.9203040Z 2022-03-17T19:17:31.9203493Z Running tests... 2022-03-17T19:17:31.9204012Z ---------------------------------------------------------------------- 2022-03-17T19:17:33.5131869Z test_create_store_multi (__main__.DistributedUtilTest) ... ok (1.593s) 2022-03-17T19:17:33.5143551Z test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s) 2022-03-17T19:17:33.5149037Z test_create_store_single_server (__main__.DistributedUtilTest) ... skip (0.000s) 2022-03-17T19:17:36.5422502Z test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.027s) 2022-03-17T19:17:36.5431683Z test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (68188c1d6584, 0). 2022-03-17T19:17:36.5433054Z ok (0.001s) 2022-03-17T19:17:36.5449954Z test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:39803 (errno: 98 - Address already in use). 2022-03-17T19:17:36.5468775Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:39803 (errno: 98 - Address already in use). 2022-03-17T19:17:36.5469258Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address. 2022-03-17T19:17:36.5472867Z ok (0.004s) 2022-03-17T19:17:36.5504449Z test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (68188c1d6584, 50493). 2022-03-17T19:17:36.5506317Z ok (0.003s) 2022-03-17T19:17:36.5506926Z 2022-03-17T19:17:36.5507258Z ---------------------------------------------------------------------- 2022-03-17T19:17:36.5507761Z Ran 7 tests in 4.630s 2022-03-17T19:17:36.5507935Z 2022-03-17T19:17:36.5508048Z OK (skipped=1) 2022-03-17T19:17:36.5508199Z 2022-03-17T19:17:36.5510997Z Generating XML reports... 2022-03-17T19:17:36.5556009Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220317191731.xml 2022-03-17T19:17:36.7867066Z Running distributed/elastic/utils/logging_test ... [2022-03-17 19:17:36.786218] 2022-03-17T19:17:36.7867780Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:36.786321] 2022-03-17T19:17:37.6478480Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test 2022-03-17T19:17:37.6500926Z 2022-03-17T19:17:37.6501420Z Running tests... 2022-03-17T19:17:37.6501825Z ---------------------------------------------------------------------- 2022-03-17T19:17:39.2420867Z test_derive_module_name (__main__.LoggingTest) ... ok (1.592s) 2022-03-17T19:17:39.2445279Z test_logger_name (__main__.LoggingTest) ... ok (0.002s) 2022-03-17T19:17:39.2445715Z 2022-03-17T19:17:39.2446260Z ---------------------------------------------------------------------- 2022-03-17T19:17:39.2446582Z Ran 2 tests in 1.594s 2022-03-17T19:17:39.2446722Z 2022-03-17T19:17:39.2446832Z OK 2022-03-17T19:17:39.2446973Z 2022-03-17T19:17:39.2447107Z Generating XML reports... 2022-03-17T19:17:39.2482440Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220317191737.xml 2022-03-17T19:17:39.4539814Z Running distributed/elastic/utils/util_test ... [2022-03-17 19:17:39.453486] 2022-03-17T19:17:39.4541125Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:39.453587] 2022-03-17T19:17:40.2427183Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test 2022-03-17T19:17:40.2450441Z 2022-03-17T19:17:40.2450581Z Running tests... 2022-03-17T19:17:40.2451431Z ---------------------------------------------------------------------- 2022-03-17T19:17:41.8041008Z test_get_data (__main__.StoreUtilTest) ... ok (1.559s) 2022-03-17T19:17:41.8058718Z test_synchronize (__main__.StoreUtilTest) ... ok (0.002s) 2022-03-17T19:17:41.8768769Z test_get_logger (__main__.UtilTest) ... ok (0.071s) 2022-03-17T19:17:41.8775550Z test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s) 2022-03-17T19:17:41.8787063Z test_get_logger_different (__main__.UtilTest) ... ok (0.001s) 2022-03-17T19:17:41.8800685Z test_get_logger_none (__main__.UtilTest) ... ok (0.001s) 2022-03-17T19:17:41.8801052Z 2022-03-17T19:17:41.8801558Z ---------------------------------------------------------------------- 2022-03-17T19:17:41.8801930Z Ran 6 tests in 1.635s 2022-03-17T19:17:41.8802099Z 2022-03-17T19:17:41.8802204Z OK 2022-03-17T19:17:41.8802341Z 2022-03-17T19:17:41.8802457Z Generating XML reports... 2022-03-17T19:17:41.8835985Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220317191740.xml 2022-03-17T19:17:41.8842410Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220317191740.xml 2022-03-17T19:17:42.0872533Z Running distributed/fsdp/test_flatten_params_wrapper ... [2022-03-17 19:17:42.086739] 2022-03-17T19:17:42.0873214Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_flatten_params_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:42.086848] 2022-03-17T19:17:42.8876900Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper 2022-03-17T19:17:42.8896743Z 2022-03-17T19:17:42.8897125Z Running tests... 2022-03-17T19:17:42.8897591Z ---------------------------------------------------------------------- 2022-03-17T19:17:44.4326551Z test_empty_module (__main__.TestFlattenParams) ... ok (1.543s) 2022-03-17T19:17:44.4422136Z test_flatten_nothing (__main__.TestFlattenParams) ... ok (0.010s) 2022-03-17T19:17:44.4528245Z test_num_params (__main__.TestFlattenParams) ... ok (0.010s) 2022-03-17T19:17:44.4800313Z test_output (__main__.TestFlattenParams) ... ok (0.027s) 2022-03-17T19:17:44.4929873Z test_partial_flattening (__main__.TestFlattenParams) ... ok (0.013s) 2022-03-17T19:17:44.5052135Z test_sharded_flat_param (__main__.TestFlattenParams) ... ok (0.012s) 2022-03-17T19:17:44.5157362Z test_shared_params_num_params (__main__.TestFlattenParams) ... ok (0.010s) 2022-03-17T19:17:44.5381323Z test_shared_params_output (__main__.TestFlattenParams) ... ok (0.022s) 2022-03-17T19:17:44.5814797Z test_shared_params_pnorm_after_step (__main__.TestFlattenParams) ... ok (0.043s) 2022-03-17T19:17:44.5829271Z test_empty_module (__main__.TestFlattenParamsCUDA) ... ok (0.001s) 2022-03-17T19:17:44.5938798Z test_flatten_nothing (__main__.TestFlattenParamsCUDA) ... ok (0.011s) 2022-03-17T19:17:44.6068562Z test_num_params (__main__.TestFlattenParamsCUDA) ... ok (0.013s) 2022-03-17T19:17:44.9128178Z test_output (__main__.TestFlattenParamsCUDA) ... ok (0.306s) 2022-03-17T19:17:44.9293162Z test_partial_flattening (__main__.TestFlattenParamsCUDA) ... ok (0.017s) 2022-03-17T19:17:44.9410958Z test_sharded_flat_param (__main__.TestFlattenParamsCUDA) ... ok (0.012s) 2022-03-17T19:17:44.9542841Z test_shared_params_num_params (__main__.TestFlattenParamsCUDA) ... ok (0.013s) 2022-03-17T19:17:44.9788610Z test_shared_params_output (__main__.TestFlattenParamsCUDA) ... ok (0.024s) 2022-03-17T19:17:45.0310005Z test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDA) ... ok (0.052s) 2022-03-17T19:17:45.0325475Z test_empty_module (__main__.TestFlattenParamsCUDAHalf) ... ok (0.001s) 2022-03-17T19:17:45.0453757Z test_flatten_nothing (__main__.TestFlattenParamsCUDAHalf) ... ok (0.013s) 2022-03-17T19:17:45.0603140Z test_num_params (__main__.TestFlattenParamsCUDAHalf) ... ok (0.015s) 2022-03-17T19:17:45.0887107Z test_output (__main__.TestFlattenParamsCUDAHalf) ... ok (0.028s) 2022-03-17T19:17:45.1070315Z test_partial_flattening (__main__.TestFlattenParamsCUDAHalf) ... ok (0.018s) 2022-03-17T19:17:45.1188330Z test_sharded_flat_param (__main__.TestFlattenParamsCUDAHalf) ... ok (0.012s) 2022-03-17T19:17:45.1339031Z test_shared_params_num_params (__main__.TestFlattenParamsCUDAHalf) ... ok (0.015s) 2022-03-17T19:17:45.1614236Z test_shared_params_output (__main__.TestFlattenParamsCUDAHalf) ... ok (0.027s) 2022-03-17T19:17:45.2189025Z test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDAHalf) ... ok (0.057s) 2022-03-17T19:17:45.2189268Z 2022-03-17T19:17:45.2189718Z ---------------------------------------------------------------------- 2022-03-17T19:17:45.2190017Z Ran 27 tests in 2.329s 2022-03-17T19:17:45.2190185Z 2022-03-17T19:17:45.2190284Z OK 2022-03-17T19:17:45.2190371Z 2022-03-17T19:17:45.2190495Z Generating XML reports... 2022-03-17T19:17:45.2233938Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220317191742.xml 2022-03-17T19:17:45.2245916Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220317191742.xml 2022-03-17T19:17:45.2257580Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220317191742.xml 2022-03-17T19:17:45.4532107Z Running distributed/fsdp/test_fsdp_apply ... [2022-03-17 19:17:45.452718] 2022-03-17T19:17:45.4532838Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:45.452824] 2022-03-17T19:17:46.2746118Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply 2022-03-17T19:17:46.2761956Z 2022-03-17T19:17:46.2762059Z Running tests... 2022-03-17T19:17:46.2763339Z ---------------------------------------------------------------------- 2022-03-17T19:17:46.2771785Z test_apply_in_summon_raises_error (__main__.TestApply) 2022-03-17T19:17:47.8135176Z Ensures that if user calls apply() on FSDP instance within full param ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:17:47.8522025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15075 2022-03-17T19:17:47.8631180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15076 2022-03-17T19:17:48.6957975Z dist init r=0, world=2 2022-03-17T19:17:48.6958302Z dist init r=1, world=2 2022-03-17T19:17:48.6961410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:17:48.6963817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:17:48.6964378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:17:48.6965280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:17:50.0987860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:17:50.0988402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:17:50.1348050Z Asserting FSDP instance is: FullyShardedDataParallel( 2022-03-17T19:17:50.1348462Z (_fsdp_wrapped_module): FlattenParamsWrapper( 2022-03-17T19:17:50.1348844Z (_fpw_module): TransformerWithSharedParams( 2022-03-17T19:17:50.1349136Z (embed_tokens): Embedding(23, 16) 2022-03-17T19:17:50.1349771Z (transformer): Transformer( 2022-03-17T19:17:50.1350078Z (encoder): TransformerEncoder( 2022-03-17T19:17:50.1350329Z (layers): ModuleList( 2022-03-17T19:17:50.1350591Z (0): TransformerEncoderLayer( 2022-03-17T19:17:50.1350905Z (self_attn): MultiheadAttention( 2022-03-17T19:17:50.1351344Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-03-17T19:17:50.1351695Z ) 2022-03-17T19:17:50.1351963Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2022-03-17T19:17:50.1352244Z (dropout): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1352648Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2022-03-17T19:17:50.1353148Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1353600Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1353971Z (dropout1): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1354343Z (dropout2): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1354611Z ) 2022-03-17T19:17:50.1354858Z (1): TransformerEncoderLayer( 2022-03-17T19:17:50.1355208Z (self_attn): MultiheadAttention( 2022-03-17T19:17:50.1355609Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-03-17T19:17:50.1355932Z ) 2022-03-17T19:17:50.1356212Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2022-03-17T19:17:50.1356562Z (dropout): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1356907Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2022-03-17T19:17:50.1357347Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1357791Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1358139Z (dropout1): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1358450Z (dropout2): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1358747Z ) 2022-03-17T19:17:50.1358968Z ) 2022-03-17T19:17:50.1359486Z (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1359968Z ) 2022-03-17T19:17:50.1360428Z (decoder): TransformerDecoder( 2022-03-17T19:17:50.1360942Z (layers): ModuleList( 2022-03-17T19:17:50.1361427Z (0): TransformerDecoderLayer( 2022-03-17T19:17:50.1362037Z (self_attn): MultiheadAttention( 2022-03-17T19:17:50.1362959Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-03-17T19:17:50.1363457Z ) 2022-03-17T19:17:50.1363733Z (multihead_attn): MultiheadAttention( 2022-03-17T19:17:50.1364131Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-03-17T19:17:50.1364477Z ) 2022-03-17T19:17:50.1364738Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2022-03-17T19:17:50.1365062Z (dropout): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1365409Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2022-03-17T19:17:50.1366033Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1366443Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1366880Z (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1367225Z (dropout1): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1367530Z (dropout2): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1367834Z (dropout3): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1368103Z ) 2022-03-17T19:17:50.1368374Z (1): TransformerDecoderLayer( 2022-03-17T19:17:50.1368734Z (self_attn): MultiheadAttention( 2022-03-17T19:17:50.1369146Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-03-17T19:17:50.1369477Z ) 2022-03-17T19:17:50.1369753Z (multihead_attn): MultiheadAttention( 2022-03-17T19:17:50.1370153Z (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True) 2022-03-17T19:17:50.1370498Z ) 2022-03-17T19:17:50.1370796Z (linear1): Linear(in_features=16, out_features=8, bias=True) 2022-03-17T19:17:50.1371116Z (dropout): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1371459Z (linear2): Linear(in_features=8, out_features=16, bias=True) 2022-03-17T19:17:50.1371900Z (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1372346Z (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1372786Z (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1373107Z (dropout1): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1373427Z (dropout2): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1392601Z (dropout3): Dropout(p=0.1, inplace=False) 2022-03-17T19:17:50.1392869Z ) 2022-03-17T19:17:50.1393103Z ) 2022-03-17T19:17:50.1393521Z (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True) 2022-03-17T19:17:50.1393855Z ) 2022-03-17T19:17:50.1394042Z ) 2022-03-17T19:17:50.1394331Z (output_proj): Linear(in_features=16, out_features=23, bias=True) 2022-03-17T19:17:50.1394854Z (bn): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) 2022-03-17T19:17:50.1395171Z ) 2022-03-17T19:17:50.1395369Z ) 2022-03-17T19:17:50.1395557Z ) 2022-03-17T19:17:50.1395945Z ERROR: expected to be in states [] but current state is TrainingState_.SUMMON_FULL_PARAMS 2022-03-17T19:17:50.1396379Z File "", line 1, in 2022-03-17T19:17:50.1396725Z File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main 2022-03-17T19:17:50.1397079Z exitcode = _main(fd) 2022-03-17T19:17:50.1397440Z File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 118, in _main 2022-03-17T19:17:50.1397780Z return self._bootstrap() 2022-03-17T19:17:50.1398157Z File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap 2022-03-17T19:17:50.1398536Z self.run() 2022-03-17T19:17:50.1398881Z File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 99, in run 2022-03-17T19:17:50.1399244Z self._target(*self._args, **self._kwargs) 2022-03-17T19:17:50.1399784Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py", line 399, in _run 2022-03-17T19:17:50.1400190Z self.run_test(test_name, pipe) 2022-03-17T19:17:50.1400721Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 601, in run_test 2022-03-17T19:17:50.1401186Z getattr(self, test_name)() 2022-03-17T19:17:50.1401686Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 486, in wrapper 2022-03-17T19:17:50.1402276Z fn() 2022-03-17T19:17:50.1402739Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 131, in wrapper 2022-03-17T19:17:50.1403133Z return func(*args, **kwargs) 2022-03-17T19:17:50.1403609Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_apply.py", line 100, in test_apply_in_summon_raises_error 2022-03-17T19:17:50.1404012Z transformer.apply(self._init_linear_weights) 2022-03-17T19:17:50.1404598Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 457, in apply 2022-03-17T19:17:50.1405051Z self._assert_state(TrainingState_.IDLE) 2022-03-17T19:17:50.1405709Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1877, in _assert_state 2022-03-17T19:17:50.1406153Z traceback.print_stack() 2022-03-17T19:17:50.3694072Z ok (4.093s) 2022-03-17T19:17:50.3701495Z test_nested_module_apply (__main__.TestApply) 2022-03-17T19:17:50.3819973Z Checks apply() modifies weights appropriately on a nested FSDP instance. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15158 2022-03-17T19:17:50.3926725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15159 2022-03-17T19:17:51.2054044Z dist init r=1, world=2 2022-03-17T19:17:51.2057290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:17:51.2339925Z dist init r=0, world=2 2022-03-17T19:17:51.2344992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:17:51.2346013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:17:51.2362305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:17:52.6085026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:17:52.6085815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:17:52.9988991Z ok (2.629s) 2022-03-17T19:17:52.9995145Z test_transformer_module_apply (__main__.TestApply) 2022-03-17T19:17:52.9999084Z Checks apply() modifies weights appropriately on a wrapped Transformer ... skip (0.001s) 2022-03-17T19:17:52.9999353Z 2022-03-17T19:17:52.9999778Z ---------------------------------------------------------------------- 2022-03-17T19:17:53.0000114Z Ran 3 tests in 6.724s 2022-03-17T19:17:53.0000281Z 2022-03-17T19:17:53.0000376Z OK (skipped=1) 2022-03-17T19:17:53.0000502Z 2022-03-17T19:17:53.0000646Z Generating XML reports... 2022-03-17T19:17:53.0046361Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220317191746.xml 2022-03-17T19:17:53.2307540Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-03-17 19:17:53.230274] 2022-03-17T19:17:53.2308285Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:17:53.230377] 2022-03-17T19:17:54.0283275Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint 2022-03-17T19:17:54.0299991Z 2022-03-17T19:17:54.0300250Z Running tests... 2022-03-17T19:17:54.0301237Z ---------------------------------------------------------------------- 2022-03-17T19:17:55.5767582Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:17:55.6152377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15278 2022-03-17T19:17:55.6257694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15279 2022-03-17T19:17:56.4566769Z dist init r=0, world=2 2022-03-17T19:17:56.4570417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:17:56.4991168Z dist init r=1, world=2 2022-03-17T19:17:56.4996379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:17:56.4997756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:17:56.5078200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:17:57.8646704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:17:57.8648011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:17:58.5328119Z ok (4.502s) 2022-03-17T19:17:58.5358490Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip (0.003s) 2022-03-17T19:17:58.5499950Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15365 2022-03-17T19:17:58.5604539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15366 2022-03-17T19:17:59.3596843Z dist init r=0, world=2 2022-03-17T19:17:59.3600161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:17:59.3669484Z dist init r=1, world=2 2022-03-17T19:17:59.3674712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:17:59.3676054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:17:59.3703287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:00.7311198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:00.7311704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:01.4669171Z ok (2.931s) 2022-03-17T19:18:01.4697713Z test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip (0.003s) 2022-03-17T19:18:01.4834297Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15452 2022-03-17T19:18:01.4940028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15453 2022-03-17T19:18:02.3148173Z dist init r=0, world=2 2022-03-17T19:18:02.3151013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:02.3629853Z dist init r=1, world=2 2022-03-17T19:18:02.3634886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:02.3636409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:02.3658958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:03.7259454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:03.7259978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:04.4004163Z ok (2.930s) 2022-03-17T19:18:04.4029055Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip (0.003s) 2022-03-17T19:18:04.4163950Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15539 2022-03-17T19:18:04.4271615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15540 2022-03-17T19:18:05.2834678Z dist init r=0, world=2 2022-03-17T19:18:05.2837656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:05.2900019Z dist init r=1, world=2 2022-03-17T19:18:05.2905069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:05.2906445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:05.2940618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:06.6477314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:06.6477867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:07.3335794Z ok (2.930s) 2022-03-17T19:18:07.3360524Z test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip (0.003s) 2022-03-17T19:18:07.3362453Z 2022-03-17T19:18:07.3363025Z ---------------------------------------------------------------------- 2022-03-17T19:18:07.3363372Z Ran 8 tests in 13.306s 2022-03-17T19:18:07.3363581Z 2022-03-17T19:18:07.3363636Z OK (skipped=4) 2022-03-17T19:18:07.3363820Z 2022-03-17T19:18:07.3363926Z Generating XML reports... 2022-03-17T19:18:07.3413478Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220317191754.xml 2022-03-17T19:18:07.5697091Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-03-17 19:18:07.569248] 2022-03-17T19:18:07.5697826Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:18:07.569352] 2022-03-17T19:18:08.3989515Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm 2022-03-17T19:18:08.4008827Z 2022-03-17T19:18:08.4009300Z Running tests... 2022-03-17T19:18:08.4009795Z ---------------------------------------------------------------------- 2022-03-17T19:18:08.4018908Z test_fsdp_calc_grad_norm_error_norm_type_1_3 (__main__.TestCalcuGradNorm) 2022-03-17T19:18:09.9401568Z Test the abnormal cases of grad norm cal API. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:18:09.9790233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15663 2022-03-17T19:18:09.9895951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15664 2022-03-17T19:18:10.7459988Z dist init r=0, world=2 2022-03-17T19:18:10.7461609Z dist init r=1, world=2 2022-03-17T19:18:10.7464636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:10.7468037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:10.7468875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:10.7568194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:12.1037873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:12.1038441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:12.6974473Z ok (4.296s) 2022-03-17T19:18:12.6984582Z test_fsdp_calc_grad_norm_error_norm_type_2_5 (__main__.TestCalcuGradNorm) 2022-03-17T19:18:12.7102349Z Test the abnormal cases of grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15750 2022-03-17T19:18:12.7212821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15751 2022-03-17T19:18:13.5200088Z dist init r=0, world=2 2022-03-17T19:18:13.5204597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:13.5498811Z dist init r=1, world=2 2022-03-17T19:18:13.5504816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:13.5505616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:13.5510252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:14.9080148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:14.9080684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:15.5280003Z ok (2.830s) 2022-03-17T19:18:15.5289847Z test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-03-17T19:18:15.5409057Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15837 2022-03-17T19:18:15.5516835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15838 2022-03-17T19:18:16.3700624Z dist init r=0, world=2 2022-03-17T19:18:16.3705439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:16.4098315Z dist init r=1, world=2 2022-03-17T19:18:16.4104641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:16.4105763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:16.4111695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:17.7635797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:17.7636771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:18.3582106Z ok (2.830s) 2022-03-17T19:18:18.3592116Z test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-03-17T19:18:18.3710612Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15924 2022-03-17T19:18:18.3816617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15925 2022-03-17T19:18:19.2149578Z dist init r=0, world=2 2022-03-17T19:18:19.2154095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:19.2400415Z dist init r=1, world=2 2022-03-17T19:18:19.2405979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:19.2407195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:19.2459776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:20.5989982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:20.5990816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:21.1880075Z ok (2.830s) 2022-03-17T19:18:21.1889382Z test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_False (__main__.TestCalcuGradNorm) 2022-03-17T19:18:21.2008019Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16011 2022-03-17T19:18:21.2113333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16012 2022-03-17T19:18:22.0080279Z dist init r=0, world=2 2022-03-17T19:18:22.0084913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:22.0551825Z dist init r=1, world=2 2022-03-17T19:18:22.0557411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:22.0558606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:22.0592132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:23.4149493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:23.4150011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:23.7230392Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:23.7231205Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:23.7240447Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:23.7241182Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:24.0175942Z ok (2.830s) 2022-03-17T19:18:24.0187242Z test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_True (__main__.TestCalcuGradNorm) 2022-03-17T19:18:24.0304295Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16098 2022-03-17T19:18:24.0412508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16099 2022-03-17T19:18:24.8557672Z dist init r=0, world=2 2022-03-17T19:18:24.8561861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:24.8944077Z dist init r=1, world=2 2022-03-17T19:18:24.8949780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:24.8950588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:24.8967725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:26.2544992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:26.2545827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:26.5684736Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:26.5685520Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:26.5686477Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:26.5687175Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:26.8476001Z ok (2.830s) 2022-03-17T19:18:26.8484689Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-03-17T19:18:26.8602380Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16185 2022-03-17T19:18:26.8708165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16186 2022-03-17T19:18:27.6978241Z dist init r=0, world=2 2022-03-17T19:18:27.6982749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:27.7352369Z dist init r=1, world=2 2022-03-17T19:18:27.7357987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:27.7359477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:27.7389335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:29.1003826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:29.1004332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:29.4042586Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:29.4043301Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:29.4057907Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:29.4058565Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:29.4088890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:29.4089575Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:29.4101313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:29.4101987Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:29.9776924Z ok (3.130s) 2022-03-17T19:18:29.9785855Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-03-17T19:18:29.9902448Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16272 2022-03-17T19:18:30.0008018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16273 2022-03-17T19:18:30.8372915Z dist init r=1, world=2 2022-03-17T19:18:30.8377533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:30.8429651Z dist init r=0, world=2 2022-03-17T19:18:30.8435371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:30.8436367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:30.8480781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:32.2152915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:32.2153802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:32.5228348Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:32.5229056Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:32.5230562Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:32.5231201Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:32.5281565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:32.5282257Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:32.5283116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:32.5283764Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:33.1076463Z ok (3.130s) 2022-03-17T19:18:33.1084770Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-03-17T19:18:33.1203670Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16359 2022-03-17T19:18:33.1309789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16360 2022-03-17T19:18:33.9889731Z dist init r=0, world=2 2022-03-17T19:18:33.9894257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:33.9963805Z dist init r=1, world=2 2022-03-17T19:18:33.9969653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:33.9970594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:33.9997062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:35.3678144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:35.3678655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:35.6752773Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:35.6753513Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:35.6757521Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:35.6758197Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:35.6817519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:35.6818497Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:35.6822037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:35.6822685Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:36.2378960Z ok (3.130s) 2022-03-17T19:18:36.2387899Z test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-03-17T19:18:36.2503695Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16446 2022-03-17T19:18:36.2611106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16447 2022-03-17T19:18:37.0634736Z dist init r=1, world=2 2022-03-17T19:18:37.0639404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:37.1043902Z dist init r=0, world=2 2022-03-17T19:18:37.1049753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:37.1050533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:37.1147154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:38.4549966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:38.4550451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:38.7622193Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:38.7622882Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:38.7665941Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:38.7666570Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:38.7745346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:38.7746036Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:38.7746880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:38.7747516Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:39.3678728Z ok (3.130s) 2022-03-17T19:18:39.3686758Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-03-17T19:18:39.3802507Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16533 2022-03-17T19:18:39.3907493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16534 2022-03-17T19:18:40.1917965Z dist init r=1, world=2 2022-03-17T19:18:40.1922317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:40.2351626Z dist init r=0, world=2 2022-03-17T19:18:40.2357264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:40.2358343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:40.2430809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:41.6154227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:41.6155031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:41.9201738Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:41.9202426Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:41.9208316Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:41.9208944Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:41.9243811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:41.9244483Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:41.9250625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:41.9251273Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:41.9278040Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:41.9278812Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:41.9279735Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:41.9280422Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:42.2073456Z ok (2.839s) 2022-03-17T19:18:42.2081775Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-03-17T19:18:42.2201680Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16620 2022-03-17T19:18:42.2307845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16621 2022-03-17T19:18:43.0397532Z dist init r=0, world=2 2022-03-17T19:18:43.0402318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:43.0729431Z dist init r=1, world=2 2022-03-17T19:18:43.0735674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:43.0736660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:43.0808823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:44.4641713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:44.4642215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:44.7687264Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:44.7687850Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:44.7725722Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:44.7726306Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:44.7776657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:44.7777245Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:44.7778131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:44.7778761Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:44.7804609Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:44.7805329Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:44.7806201Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:44.7806882Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:45.0370499Z ok (2.830s) 2022-03-17T19:18:45.0378451Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm) 2022-03-17T19:18:45.0497769Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16707 2022-03-17T19:18:45.0603368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16708 2022-03-17T19:18:45.8976786Z dist init r=1, world=2 2022-03-17T19:18:45.8981398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:45.9154190Z dist init r=0, world=2 2022-03-17T19:18:45.9159886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:45.9160681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:45.9186006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:47.2920994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:47.2921479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:47.5976089Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:47.5976781Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:47.6016845Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:47.6017539Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:47.6076142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:47.6076815Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:47.6081295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:47.6081905Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:47.6108589Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:47.6109265Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:47.6111267Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:47.6111945Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:47.8667851Z ok (2.830s) 2022-03-17T19:18:47.8676005Z test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm) 2022-03-17T19:18:47.8794257Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16794 2022-03-17T19:18:47.8897988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16795 2022-03-17T19:18:48.7484452Z dist init r=1, world=2 2022-03-17T19:18:48.7489035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:48.7493529Z dist init r=0, world=2 2022-03-17T19:18:48.7499460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:48.7500586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:48.7591911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:50.1197178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:50.1197752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:50.4201300Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:50.4201996Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:50.4285510Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_clip_grad_norm.py:51: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:50.4286169Z in_data = torch.tensor(input[self.rank], device=self.rank) 2022-03-17T19:18:50.4362414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:50.4363054Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:50.4363925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py:608: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:50.4364577Z return_norm = torch.tensor(total_norm ** norm_type, device=rank) 2022-03-17T19:18:50.4391288Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:50.4392105Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:50.4393070Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2030: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). 2022-03-17T19:18:50.4393772Z local_norm = torch.tensor(max(par.grad.detach().abs().max() for par in parameters)) 2022-03-17T19:18:50.6961357Z ok (2.829s) 2022-03-17T19:18:50.6961627Z 2022-03-17T19:18:50.6962043Z ---------------------------------------------------------------------- 2022-03-17T19:18:50.6962337Z Ran 14 tests in 42.295s 2022-03-17T19:18:50.6965068Z 2022-03-17T19:18:50.6965484Z OK 2022-03-17T19:18:50.6965656Z 2022-03-17T19:18:50.6965803Z Generating XML reports... 2022-03-17T19:18:50.7013524Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220317191808.xml 2022-03-17T19:18:50.7025464Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220317191808.xml 2022-03-17T19:18:50.9323327Z Running distributed/fsdp/test_fsdp_comm ... [2022-03-17 19:18:50.931849] 2022-03-17T19:18:50.9324006Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:18:50.931955] 2022-03-17T19:18:51.7070457Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm 2022-03-17T19:18:51.7088112Z 2022-03-17T19:18:51.7088345Z Running tests... 2022-03-17T19:18:51.7089117Z ---------------------------------------------------------------------- 2022-03-17T19:18:51.7129088Z test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication) 2022-03-17T19:18:53.2602135Z Tests FSDP's communication cost in terms of calls to collective ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:18:53.2990826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16918 2022-03-17T19:18:53.3100843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16919 2022-03-17T19:18:54.1098616Z dist init r=0, world=2 2022-03-17T19:18:54.1101728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:54.1207929Z dist init r=1, world=2 2022-03-17T19:18:54.1212626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:54.1213961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:54.1306275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:55.5052754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:55.5053671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:56.6177657Z ok (4.909s) 2022-03-17T19:18:56.6216440Z test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-03-17T19:18:56.6336634Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17005 2022-03-17T19:18:56.6443159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17006 2022-03-17T19:18:57.5013387Z dist init r=0, world=2 2022-03-17T19:18:57.5017533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:18:57.5038279Z dist init r=1, world=2 2022-03-17T19:18:57.5043421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:18:57.5044240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:57.5120229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:18:58.8816238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:18:58.8816770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:18:59.9518476Z ok (3.334s) 2022-03-17T19:18:59.9555955Z test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication) 2022-03-17T19:18:59.9676581Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17092 2022-03-17T19:18:59.9783637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17093 2022-03-17T19:19:00.8171463Z dist init r=0, world=2 2022-03-17T19:19:00.8174990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:00.8274266Z dist init r=1, world=2 2022-03-17T19:19:00.8278862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:00.8280221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:00.8379744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:02.2277428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:02.2277942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:03.3860802Z ok (3.434s) 2022-03-17T19:19:03.3898843Z test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-03-17T19:19:03.4016933Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17179 2022-03-17T19:19:03.4122785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17180 2022-03-17T19:19:04.2235038Z dist init r=0, world=2 2022-03-17T19:19:04.2238630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:04.2256074Z dist init r=1, world=2 2022-03-17T19:19:04.2261085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:04.2262255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:04.2341865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:05.6258897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:05.6259360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:06.8197787Z ok (3.434s) 2022-03-17T19:19:06.8235562Z test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication) 2022-03-17T19:19:06.8354085Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17266 2022-03-17T19:19:06.8460760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17267 2022-03-17T19:19:07.6623112Z dist init r=1, world=2 2022-03-17T19:19:07.6627402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:07.7265770Z dist init r=0, world=2 2022-03-17T19:19:07.7270646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:07.7271644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:07.7338011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:09.1083994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:09.1084939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:09.7526476Z ok (2.933s) 2022-03-17T19:19:09.7565102Z test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-03-17T19:19:09.7683805Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17353 2022-03-17T19:19:09.7791303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17354 2022-03-17T19:19:10.5811418Z dist init r=0, world=2 2022-03-17T19:19:10.5814317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:10.6198275Z dist init r=1, world=2 2022-03-17T19:19:10.6202774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:10.6204193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:10.6220647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:12.0043659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:12.0044144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:12.5855356Z ok (2.833s) 2022-03-17T19:19:12.5892949Z test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication) 2022-03-17T19:19:12.6010865Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17440 2022-03-17T19:19:12.6118628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17441 2022-03-17T19:19:13.4158079Z dist init r=0, world=2 2022-03-17T19:19:13.4161115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:13.4225256Z dist init r=1, world=2 2022-03-17T19:19:13.4229939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:13.4231037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:13.4264430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:14.8014050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:14.8014575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:15.4186275Z ok (2.833s) 2022-03-17T19:19:15.4223385Z test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication) 2022-03-17T19:19:15.4341812Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17527 2022-03-17T19:19:15.4447956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17528 2022-03-17T19:19:16.2877385Z dist init r=1, world=2 2022-03-17T19:19:16.2880933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:16.3045708Z dist init r=0, world=2 2022-03-17T19:19:16.3050284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:16.3051005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:16.3085102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:17.6700488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:17.6701020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:18.3515518Z ok (2.933s) 2022-03-17T19:19:18.3515914Z 2022-03-17T19:19:18.3516629Z ---------------------------------------------------------------------- 2022-03-17T19:19:18.3517246Z Ran 8 tests in 26.643s 2022-03-17T19:19:18.3518085Z 2022-03-17T19:19:18.3518441Z OK 2022-03-17T19:19:18.3518639Z 2022-03-17T19:19:18.3518786Z Generating XML reports... 2022-03-17T19:19:18.3574564Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20220317191851.xml 2022-03-17T19:19:18.5843347Z Running distributed/fsdp/test_fsdp_core ... [2022-03-17 19:19:18.583811] 2022-03-17T19:19:18.5844124Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:19:18.583920] 2022-03-17T19:19:19.3432759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0vm2va2 2022-03-17T19:19:19.3433517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0vm2va2/_remote_module_non_sriptable.py 2022-03-17T19:19:19.3583019Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core 2022-03-17T19:19:19.3619436Z 2022-03-17T19:19:19.3619684Z Running tests... 2022-03-17T19:19:19.3620209Z ---------------------------------------------------------------------- 2022-03-17T19:19:20.8846282Z test_backward_hooks_after_save (__main__.TestHooks) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:19:20.9233606Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17651 2022-03-17T19:19:20.9345234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17652 2022-03-17T19:19:21.7310712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7pzatrd 2022-03-17T19:19:21.7311520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7pzatrd/_remote_module_non_sriptable.py 2022-03-17T19:19:21.7358338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv33m1cb 2022-03-17T19:19:21.7361792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv33m1cb/_remote_module_non_sriptable.py 2022-03-17T19:19:21.7414477Z dist init r=1, world=2 2022-03-17T19:19:21.7419001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:21.7467123Z dist init r=0, world=2 2022-03-17T19:19:21.7471706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:21.7472576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:21.7522229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:23.1360079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:23.1361018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:23.8430004Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:19:23.8431713Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:19:23.8436732Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:19:23.8438276Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:19:24.2421553Z ok (4.880s) 2022-03-17T19:19:24.2545788Z test_output_backward_hooks_cuda_first_False (__main__.TestHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17738 2022-03-17T19:19:24.2650591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17739 2022-03-17T19:19:25.0674437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezgcag9u 2022-03-17T19:19:25.0678523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezgcag9u/_remote_module_non_sriptable.py 2022-03-17T19:19:25.0692179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpecavlu9e 2022-03-17T19:19:25.0692754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpecavlu9e/_remote_module_non_sriptable.py 2022-03-17T19:19:25.0784158Z dist init r=0, world=2 2022-03-17T19:19:25.0788070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:25.0798624Z dist init r=1, world=2 2022-03-17T19:19:25.0803496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:25.0804277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:25.0891358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:26.4333188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:26.4333729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:27.4723825Z ok (3.230s) 2022-03-17T19:19:27.4851302Z test_output_backward_hooks_cuda_first_True (__main__.TestHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17825 2022-03-17T19:19:27.4954470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17826 2022-03-17T19:19:28.2918091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwobtl9ke 2022-03-17T19:19:28.2919228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwobtl9ke/_remote_module_non_sriptable.py 2022-03-17T19:19:28.2979759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_geakbp 2022-03-17T19:19:28.2982441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_geakbp/_remote_module_non_sriptable.py 2022-03-17T19:19:28.3027694Z dist init r=1, world=2 2022-03-17T19:19:28.3032382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:28.3087050Z dist init r=0, world=2 2022-03-17T19:19:28.3091481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:28.3092245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:28.3135808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:29.6725648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:29.6726172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:30.7026058Z ok (3.230s) 2022-03-17T19:19:30.7041019Z test_register_functions_called_cuda_first_False (__main__.TestHooks) 2022-03-17T19:19:30.7158832Z Tests that _register_{pre|post}_backward_hooks called during forward. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17912 2022-03-17T19:19:30.7264836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17913 2022-03-17T19:19:31.5646369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnsde6tc 2022-03-17T19:19:31.5647654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnsde6tc/_remote_module_non_sriptable.py 2022-03-17T19:19:31.5733655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhnno_3_ 2022-03-17T19:19:31.5735928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhnno_3_/_remote_module_non_sriptable.py 2022-03-17T19:19:31.5753455Z dist init r=0, world=2 2022-03-17T19:19:31.5757932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:31.5838335Z dist init r=1, world=2 2022-03-17T19:19:31.5842567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:31.5843708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:31.5861009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:32.9440387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:32.9441460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:34.0337199Z ok (3.331s) 2022-03-17T19:19:34.0349111Z test_register_functions_called_cuda_first_True (__main__.TestHooks) 2022-03-17T19:19:34.0468128Z Tests that _register_{pre|post}_backward_hooks called during forward. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17995 2022-03-17T19:19:34.0573647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17996 2022-03-17T19:19:34.8495919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_votk4mf 2022-03-17T19:19:34.8496937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_votk4mf/_remote_module_non_sriptable.py 2022-03-17T19:19:34.8604011Z dist init r=0, world=2 2022-03-17T19:19:34.8608466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:34.9045897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0215yc0l 2022-03-17T19:19:34.9048217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0215yc0l/_remote_module_non_sriptable.py 2022-03-17T19:19:34.9151572Z dist init r=1, world=2 2022-03-17T19:19:34.9156073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:34.9156874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:34.9218062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:36.2915773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:36.2916295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:37.3645819Z ok (3.331s) 2022-03-17T19:19:37.3775411Z test_transformer_no_grad (__main__.TestNoGrad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18078 2022-03-17T19:19:37.3880889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18079 2022-03-17T19:19:38.1881071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb54g5hpn 2022-03-17T19:19:38.1881935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb54g5hpn/_remote_module_non_sriptable.py 2022-03-17T19:19:38.1906322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ot5sshu 2022-03-17T19:19:38.1908942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ot5sshu/_remote_module_non_sriptable.py 2022-03-17T19:19:38.1986225Z dist init r=1, world=2 2022-03-17T19:19:38.1990554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:38.2013538Z dist init r=0, world=2 2022-03-17T19:19:38.2017672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:38.2018976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:38.2093976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:39.5797508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:39.5798011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:40.6954461Z ok (3.331s) 2022-03-17T19:19:40.7086398Z test_param_change_after_init (__main__.TestParamInit) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18165 2022-03-17T19:19:40.7191309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18166 2022-03-17T19:19:41.5271178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_em7w1b 2022-03-17T19:19:41.5272052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_em7w1b/_remote_module_non_sriptable.py 2022-03-17T19:19:41.5378457Z dist init r=1, world=2 2022-03-17T19:19:41.5383464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:41.5558705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaznek293 2022-03-17T19:19:41.5561432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaznek293/_remote_module_non_sriptable.py 2022-03-17T19:19:41.5664927Z dist init r=0, world=2 2022-03-17T19:19:41.5669352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:41.5670421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:41.5688988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:42.9370698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:42.9371211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:43.9262025Z ok (3.231s) 2022-03-17T19:19:43.9389769Z test_delayed_optim_step_offload_false_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18248 2022-03-17T19:19:43.9494449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18249 2022-03-17T19:19:44.8060068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy74ejtgc 2022-03-17T19:19:44.8061237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy74ejtgc/_remote_module_non_sriptable.py 2022-03-17T19:19:44.8068790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpf4npqi 2022-03-17T19:19:44.8071677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpf4npqi/_remote_module_non_sriptable.py 2022-03-17T19:19:44.8166512Z dist init r=1, world=2 2022-03-17T19:19:44.8171050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:44.8176830Z dist init r=0, world=2 2022-03-17T19:19:44.8181593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:44.8182640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:44.8274605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:46.2177033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:46.2177547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:46.7840003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:46.7840509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:47.1774105Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:19:47.1774985Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:19:47.1777194Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:19:47.1778004Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:19:47.3756797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:19:47.3757767Z warnings.warn(msg, FutureWarning) 2022-03-17T19:19:47.3758690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:19:47.3759350Z warnings.warn(msg, FutureWarning) 2022-03-17T19:19:47.5298348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:47.5298995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:48.2577192Z ok (4.331s) 2022-03-17T19:19:48.2702576Z test_delayed_optim_step_offload_false_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18335 2022-03-17T19:19:48.2806828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18336 2022-03-17T19:19:49.0980006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlarig38 2022-03-17T19:19:49.0981743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlarig38/_remote_module_non_sriptable.py 2022-03-17T19:19:49.1087301Z dist init r=1, world=2 2022-03-17T19:19:49.1091765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:49.1222311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsh215b5 2022-03-17T19:19:49.1225725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsh215b5/_remote_module_non_sriptable.py 2022-03-17T19:19:49.1330830Z dist init r=0, world=2 2022-03-17T19:19:49.1335940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:49.1336963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:49.1397640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:50.5031785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:50.5032290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:51.0697089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:51.0697664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:51.5724359Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:19:51.5725189Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:19:51.5726212Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:19:51.5726973Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:19:51.8571324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:19:51.8572334Z warnings.warn(msg, FutureWarning) 2022-03-17T19:19:51.8573283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:19:51.8573947Z warnings.warn(msg, FutureWarning) 2022-03-17T19:19:52.1126919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:52.1127432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:53.1902130Z ok (4.932s) 2022-03-17T19:19:53.2027742Z test_delayed_optim_step_offload_false_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18422 2022-03-17T19:19:53.2131782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18423 2022-03-17T19:19:54.0632956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbb8wcu2 2022-03-17T19:19:54.0634127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbb8wcu2/_remote_module_non_sriptable.py 2022-03-17T19:19:54.0640604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67j2oi8p 2022-03-17T19:19:54.0643424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67j2oi8p/_remote_module_non_sriptable.py 2022-03-17T19:19:54.0739150Z dist init r=1, world=2 2022-03-17T19:19:54.0743681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:54.0749733Z dist init r=0, world=2 2022-03-17T19:19:54.0754070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:54.0755479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:54.0847159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:55.4641641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:19:55.4642166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:19:56.0309713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:56.0310248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:56.5337788Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:19:56.5338629Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:19:56.5339638Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:19:56.5340626Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:19:56.8190035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:19:56.8190694Z warnings.warn(msg, FutureWarning) 2022-03-17T19:19:56.8192241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:19:56.8192896Z warnings.warn(msg, FutureWarning) 2022-03-17T19:19:57.0748166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:57.0748661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:19:58.1226223Z ok (4.932s) 2022-03-17T19:19:58.1368614Z test_delayed_optim_step_offload_false_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18509 2022-03-17T19:19:58.1456305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18510 2022-03-17T19:19:58.9381381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlagzrqq 2022-03-17T19:19:58.9382384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlagzrqq/_remote_module_non_sriptable.py 2022-03-17T19:19:58.9504202Z dist init r=1, world=2 2022-03-17T19:19:58.9509033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:19:58.9651284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7pjx0ms 2022-03-17T19:19:58.9653993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7pjx0ms/_remote_module_non_sriptable.py 2022-03-17T19:19:58.9756225Z dist init r=0, world=2 2022-03-17T19:19:58.9760349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:19:58.9761407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:19:58.9815368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:00.3367878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:00.3368395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:00.8982645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:00.8983126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:01.4020770Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:01.4021595Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:01.4022658Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:01.4023426Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:01.6870916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:01.6871643Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:01.6872570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:01.6873529Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:01.9443729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:01.9444236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:03.0551662Z ok (4.932s) 2022-03-17T19:20:03.0678047Z test_delayed_optim_step_offload_false_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18596 2022-03-17T19:20:03.0780790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18597 2022-03-17T19:20:03.8747873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08pf42ne 2022-03-17T19:20:03.8749317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08pf42ne/_remote_module_non_sriptable.py 2022-03-17T19:20:03.8856387Z dist init r=1, world=2 2022-03-17T19:20:03.8861404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:03.9275560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1hfqm2q 2022-03-17T19:20:03.9278123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1hfqm2q/_remote_module_non_sriptable.py 2022-03-17T19:20:03.9381956Z dist init r=0, world=2 2022-03-17T19:20:03.9386844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:03.9387803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:03.9470799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:05.3149154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:05.3150092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:05.9162981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:05.9163952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:06.4193729Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:06.4195235Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:06.4197257Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:06.4198747Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:06.7047465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:06.7048879Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:06.7050742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:06.7052323Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:06.9603260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:06.9604194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:08.0877691Z ok (5.032s) 2022-03-17T19:20:08.1004061Z test_delayed_optim_step_offload_false_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18683 2022-03-17T19:20:08.1109571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18684 2022-03-17T19:20:08.9034761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrhud839 2022-03-17T19:20:08.9036597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrhud839/_remote_module_non_sriptable.py 2022-03-17T19:20:08.9156869Z dist init r=1, world=2 2022-03-17T19:20:08.9161038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:08.9170282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0qjbxcc 2022-03-17T19:20:08.9173439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0qjbxcc/_remote_module_non_sriptable.py 2022-03-17T19:20:08.9277811Z dist init r=0, world=2 2022-03-17T19:20:08.9281677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:08.9282764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:08.9366046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:10.2902034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:10.2902610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:10.8673701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:10.8674242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:11.2708271Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:11.2709030Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:11.2710054Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:11.2710831Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:11.4710647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:11.4711318Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:11.4714072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:11.4714951Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:11.6266869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:11.6267347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:12.4195914Z ok (4.332s) 2022-03-17T19:20:12.4324217Z test_delayed_optim_step_offload_true_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18770 2022-03-17T19:20:12.4428780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18771 2022-03-17T19:20:13.2408163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei62ujk3 2022-03-17T19:20:13.2409089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei62ujk3/_remote_module_non_sriptable.py 2022-03-17T19:20:13.2514970Z dist init r=1, world=2 2022-03-17T19:20:13.2520167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:13.2850756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyo4q_6y3 2022-03-17T19:20:13.2852877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyo4q_6y3/_remote_module_non_sriptable.py 2022-03-17T19:20:13.2959016Z dist init r=0, world=2 2022-03-17T19:20:13.2964233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:13.2965564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:13.3028540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:14.6569484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:14.6570454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:15.2197298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:15.2197786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:15.7201933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:15.7202413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:15.9793586Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:15.9794906Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:16.2381638Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:16.2382428Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:16.2385124Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:16.2385904Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:16.5353586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:16.5354300Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:16.5355887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:16.5356543Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:16.7909095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:16.7909574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:17.8528476Z ok (5.433s) 2022-03-17T19:20:17.8658743Z test_delayed_optim_step_offload_true_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18857 2022-03-17T19:20:17.8763492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18858 2022-03-17T19:20:18.6719791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbis9_se5 2022-03-17T19:20:18.6720904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbis9_se5/_remote_module_non_sriptable.py 2022-03-17T19:20:18.6826119Z dist init r=0, world=2 2022-03-17T19:20:18.6830508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:18.7295811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45fatn_g 2022-03-17T19:20:18.7298400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45fatn_g/_remote_module_non_sriptable.py 2022-03-17T19:20:18.7403028Z dist init r=1, world=2 2022-03-17T19:20:18.7407785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:18.7408865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:18.7439556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:20.1202526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:20.1203091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:20.6854022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:20.6854545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:21.1861310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:21.1861832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:21.4448781Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:21.4450107Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:21.7033285Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:21.7034279Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:21.7035512Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:21.7036283Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:21.9992114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:21.9992831Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:21.9993799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:21.9994396Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:22.2547410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:22.2547908Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:23.3878888Z ok (5.535s) 2022-03-17T19:20:23.4005006Z test_delayed_optim_step_offload_true_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18944 2022-03-17T19:20:23.4108909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18945 2022-03-17T19:20:24.2243295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhlfk3pm 2022-03-17T19:20:24.2244365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhlfk3pm/_remote_module_non_sriptable.py 2022-03-17T19:20:24.2351344Z dist init r=0, world=2 2022-03-17T19:20:24.2355774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:24.2381962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo9g1rhq 2022-03-17T19:20:24.2384995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo9g1rhq/_remote_module_non_sriptable.py 2022-03-17T19:20:24.2487063Z dist init r=1, world=2 2022-03-17T19:20:24.2491613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:24.2492809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:24.2560639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:25.6146130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:25.6146579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:26.1825206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:26.1825725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:26.6824859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:26.6825358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:26.9414296Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:26.9416056Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:27.2007961Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:27.2008770Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:27.2009792Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:27.2010544Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:27.4982644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:27.4983359Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:27.4984966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:27.4985543Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:27.7537689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:27.7538186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:28.8209479Z ok (5.433s) 2022-03-17T19:20:28.8337574Z test_delayed_optim_step_offload_true_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19031 2022-03-17T19:20:28.8443103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19032 2022-03-17T19:20:29.6990290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ffy2pw0 2022-03-17T19:20:29.6991537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ffy2pw0/_remote_module_non_sriptable.py 2022-03-17T19:20:29.7005515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy41hd99 2022-03-17T19:20:29.7008551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy41hd99/_remote_module_non_sriptable.py 2022-03-17T19:20:29.7099586Z dist init r=1, world=2 2022-03-17T19:20:29.7104474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:29.7116287Z dist init r=0, world=2 2022-03-17T19:20:29.7120924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:29.7121792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:29.7207804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:31.0805345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:31.0805922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:31.6484380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:31.6484823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:32.1488796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:32.1489226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:32.4074385Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:32.4075707Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:32.6656031Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:32.6656820Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:32.6658745Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:32.6659513Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:32.9617127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:32.9617735Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:32.9619255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:32.9619856Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:33.2173509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:33.2174264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:34.3545481Z ok (5.533s) 2022-03-17T19:20:34.3673978Z test_delayed_optim_step_offload_true_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19118 2022-03-17T19:20:34.3779845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19119 2022-03-17T19:20:35.1663511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0uxsg4_ 2022-03-17T19:20:35.1664419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0uxsg4_/_remote_module_non_sriptable.py 2022-03-17T19:20:35.1767292Z dist init r=1, world=2 2022-03-17T19:20:35.1771496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:35.1836046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetf_62k0 2022-03-17T19:20:35.1838699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetf_62k0/_remote_module_non_sriptable.py 2022-03-17T19:20:35.1942600Z dist init r=0, world=2 2022-03-17T19:20:35.1946980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:35.1948089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:35.1975635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:36.5560743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:36.5561283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:37.1242520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:37.1242987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:37.6244593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:37.6245103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:37.8832694Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:37.8834309Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:38.1427821Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:38.1428647Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:38.1429672Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:38.1430446Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:38.4392642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:38.4393335Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:38.4396503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:38.4397363Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:38.6947420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:38.6947911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:39.7879977Z ok (5.433s) 2022-03-17T19:20:39.8009530Z test_delayed_optim_step_offload_true_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19205 2022-03-17T19:20:39.8115830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19206 2022-03-17T19:20:40.6806074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsosjae7 2022-03-17T19:20:40.6807068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsosjae7/_remote_module_non_sriptable.py 2022-03-17T19:20:40.6868566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwpan0_r 2022-03-17T19:20:40.6871910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwpan0_r/_remote_module_non_sriptable.py 2022-03-17T19:20:40.6914306Z dist init r=1, world=2 2022-03-17T19:20:40.6918787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:40.6974894Z dist init r=0, world=2 2022-03-17T19:20:40.6979264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:40.6980614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:40.7021953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:42.0570666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:42.0571194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:42.6260754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:42.6261521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:43.1266229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:43.1266954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:43.3847806Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:43.3850200Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:20:43.6437425Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:43.6438408Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:43.6439708Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:43.6440499Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:43.9392559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:43.9393253Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:43.9397061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:43.9397766Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:44.1948329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:44.1948802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:45.3217045Z ok (5.534s) 2022-03-17T19:20:45.3342634Z test_delayed_reduce_scatter_offload_false_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19292 2022-03-17T19:20:45.3447194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19293 2022-03-17T19:20:46.1648224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxn7v_aq 2022-03-17T19:20:46.1649294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxn7v_aq/_remote_module_non_sriptable.py 2022-03-17T19:20:46.1753267Z dist init r=0, world=2 2022-03-17T19:20:46.1757619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:46.1834371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6rdhv38 2022-03-17T19:20:46.1837350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6rdhv38/_remote_module_non_sriptable.py 2022-03-17T19:20:46.1942661Z dist init r=1, world=2 2022-03-17T19:20:46.1947037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:46.1948055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:46.1961651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:47.5602301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:47.5602796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:47.8714454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:47.8723885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:48.8867422Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:48.8868261Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:48.8869301Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:48.8870058Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:49.9163354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:49.9164091Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:49.9165806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:49.9166671Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:49.9257885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:49.9259154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:52.2568229Z ok (6.935s) 2022-03-17T19:20:52.2697357Z test_delayed_reduce_scatter_offload_false_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19379 2022-03-17T19:20:52.2802401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19380 2022-03-17T19:20:53.0799917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ogkzo_3 2022-03-17T19:20:53.0800790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ogkzo_3/_remote_module_non_sriptable.py 2022-03-17T19:20:53.0906189Z dist init r=1, world=2 2022-03-17T19:20:53.0910662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:20:53.1472701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjr9z8ag 2022-03-17T19:20:53.1475522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjr9z8ag/_remote_module_non_sriptable.py 2022-03-17T19:20:53.1579947Z dist init r=0, world=2 2022-03-17T19:20:53.1585640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:20:53.1586392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:53.1620633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:20:54.5182878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:20:54.5183427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:20:54.8348489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:54.8349009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:55.8481880Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:55.8482691Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:55.8483725Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:20:55.8484492Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:20:56.8767043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:56.8767940Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:56.8770187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:20:56.8770862Z warnings.warn(msg, FutureWarning) 2022-03-17T19:20:56.8861273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:56.8861799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:20:59.1924110Z ok (6.935s) 2022-03-17T19:20:59.2050577Z test_delayed_reduce_scatter_offload_false_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19466 2022-03-17T19:20:59.2155059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19467 2022-03-17T19:21:00.0230283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1k1fez2 2022-03-17T19:21:00.0231306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1k1fez2/_remote_module_non_sriptable.py 2022-03-17T19:21:00.0336395Z dist init r=1, world=2 2022-03-17T19:21:00.0340979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:00.0614322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0r9cs8w7 2022-03-17T19:21:00.0617098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0r9cs8w7/_remote_module_non_sriptable.py 2022-03-17T19:21:00.0719390Z dist init r=0, world=2 2022-03-17T19:21:00.0723582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:00.0724407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:00.0747916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:01.4469146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:01.4469670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:01.7672444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:01.7673033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:02.6136845Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:02.6137650Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:02.6138692Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:02.6139458Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:03.2430800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:03.2431687Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:03.2432603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:03.2433267Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:03.2524132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:03.2524736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:04.7257136Z ok (5.533s) 2022-03-17T19:21:04.7386132Z test_delayed_reduce_scatter_offload_false_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19553 2022-03-17T19:21:04.7491058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19554 2022-03-17T19:21:05.5506948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85w4v01_ 2022-03-17T19:21:05.5508404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85w4v01_/_remote_module_non_sriptable.py 2022-03-17T19:21:05.5612233Z dist init r=1, world=2 2022-03-17T19:21:05.5617199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:05.5867845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclhqo304 2022-03-17T19:21:05.5870766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclhqo304/_remote_module_non_sriptable.py 2022-03-17T19:21:05.5974936Z dist init r=0, world=2 2022-03-17T19:21:05.5979653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:05.5981376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:05.6024214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:06.9549274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:06.9550105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:07.2645839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:07.2647225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:08.2783879Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:08.2784686Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:08.2785731Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:08.2786491Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:09.3062678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:09.3064368Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:09.3065492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:09.3066153Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:09.3158462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:11.6615673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:11.6616150Z ok (6.936s) 2022-03-17T19:21:11.6746466Z test_delayed_reduce_scatter_offload_false_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19640 2022-03-17T19:21:11.6852094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19641 2022-03-17T19:21:12.5353411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij0h5383 2022-03-17T19:21:12.5354526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij0h5383/_remote_module_non_sriptable.py 2022-03-17T19:21:12.5355116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1or2o4b 2022-03-17T19:21:12.5357887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1or2o4b/_remote_module_non_sriptable.py 2022-03-17T19:21:12.5458947Z dist init r=0, world=2 2022-03-17T19:21:12.5463939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:12.5464330Z dist init r=1, world=2 2022-03-17T19:21:12.5469587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:12.5470747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:12.5567353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:13.9168152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:13.9168679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:14.2652194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:14.2652711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:15.2801035Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:15.2801814Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:15.2802873Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:15.2803633Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:16.3096576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:16.3097280Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:16.3099583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:16.3100685Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:16.3193637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:16.3194119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:18.6978145Z ok (7.036s) 2022-03-17T19:21:18.7107833Z test_delayed_reduce_scatter_offload_false_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19727 2022-03-17T19:21:18.7214585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19728 2022-03-17T19:21:19.5304907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4c1dv2g3 2022-03-17T19:21:19.5306234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4c1dv2g3/_remote_module_non_sriptable.py 2022-03-17T19:21:19.5310740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2_rua8t 2022-03-17T19:21:19.5313925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2_rua8t/_remote_module_non_sriptable.py 2022-03-17T19:21:19.5410351Z dist init r=0, world=2 2022-03-17T19:21:19.5414582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:19.5433462Z dist init r=1, world=2 2022-03-17T19:21:19.5438522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:19.5439834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:19.5517967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:20.8948328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:20.8948884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:21.2065534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:21.2072173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:22.2210211Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:22.2211034Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:22.2212071Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:22.2212839Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:23.2495380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:23.2496032Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:23.2496933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:23.2497811Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:23.2586124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:23.2586570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:25.6334797Z ok (6.936s) 2022-03-17T19:21:25.6466366Z test_delayed_reduce_scatter_offload_true_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19814 2022-03-17T19:21:25.6571003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19815 2022-03-17T19:21:26.4620322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesm2sh38 2022-03-17T19:21:26.4621946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesm2sh38/_remote_module_non_sriptable.py 2022-03-17T19:21:26.4683951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x1dzq2d 2022-03-17T19:21:26.4686349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x1dzq2d/_remote_module_non_sriptable.py 2022-03-17T19:21:26.4727146Z dist init r=0, world=2 2022-03-17T19:21:26.4731501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:26.4791864Z dist init r=1, world=2 2022-03-17T19:21:26.4796436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:26.4797266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:26.4834928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:27.8478132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:27.8478669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:28.1605506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:28.1615614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:28.1731616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:28.1732109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:28.1859019Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:21:29.1913455Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:21:29.1914389Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T19:21:29.1933354Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:29.1934072Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:29.1935663Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:29.1936469Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:30.2292140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:30.2292851Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:30.2294114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:30.2294730Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:30.2384045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:30.2384564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:32.5693914Z ok (6.936s) 2022-03-17T19:21:32.5824403Z test_delayed_reduce_scatter_offload_true_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19901 2022-03-17T19:21:32.5929321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19902 2022-03-17T19:21:33.3947905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ho78l4a 2022-03-17T19:21:33.3949680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ho78l4a/_remote_module_non_sriptable.py 2022-03-17T19:21:33.3989140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzveexame 2022-03-17T19:21:33.3991728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzveexame/_remote_module_non_sriptable.py 2022-03-17T19:21:33.4055331Z dist init r=1, world=2 2022-03-17T19:21:33.4060237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:33.4096303Z dist init r=0, world=2 2022-03-17T19:21:33.4100334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:33.4101581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:33.4163074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:34.7736920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:34.7737428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:35.0852185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:35.0852706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:35.0970569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:35.0971100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:35.1095062Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:21:36.1147410Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:21:36.1148319Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T19:21:36.1167079Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:36.1167888Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:36.1168892Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:36.1169640Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:37.1523500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:37.1524179Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:37.1525676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:37.1526328Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:37.1619120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:37.1619649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:39.5050547Z ok (6.936s) 2022-03-17T19:21:39.5178033Z test_delayed_reduce_scatter_offload_true_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19988 2022-03-17T19:21:39.5282847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19989 2022-03-17T19:21:40.3238362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tl9dk5g 2022-03-17T19:21:40.3239604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tl9dk5g/_remote_module_non_sriptable.py 2022-03-17T19:21:40.3346017Z dist init r=1, world=2 2022-03-17T19:21:40.3350246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:40.3838343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbgxc2b1 2022-03-17T19:21:40.3841303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbgxc2b1/_remote_module_non_sriptable.py 2022-03-17T19:21:40.3946874Z dist init r=0, world=2 2022-03-17T19:21:40.3951577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:40.3952350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:40.3959473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:41.7668455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:41.7668978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:42.0780412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:42.0790481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:42.0909299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:42.0909757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:42.1038612Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:21:43.1101228Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:21:43.1102188Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T19:21:43.1123452Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:43.1124253Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:43.1125259Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:43.1126040Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:44.1492695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:44.1493675Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:44.1494625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:44.1495294Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:44.1588133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:44.1588633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:46.5421888Z ok (7.037s) 2022-03-17T19:21:46.5551894Z test_delayed_reduce_scatter_offload_true_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20075 2022-03-17T19:21:46.5658901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20076 2022-03-17T19:21:47.4109756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvw3mdai7 2022-03-17T19:21:47.4111154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvw3mdai7/_remote_module_non_sriptable.py 2022-03-17T19:21:47.4111871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp674tnhqa 2022-03-17T19:21:47.4114923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp674tnhqa/_remote_module_non_sriptable.py 2022-03-17T19:21:47.4215343Z dist init r=1, world=2 2022-03-17T19:21:47.4219734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:47.4220089Z dist init r=0, world=2 2022-03-17T19:21:47.4225916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:47.4226744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:47.4322987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:48.8084398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:48.8084920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:49.1585132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:49.1585614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:49.1704494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:49.1705205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:49.1825890Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:21:49.9988169Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:21:49.9989099Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T19:21:50.0009819Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:50.0010627Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:50.0011650Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:50.0012411Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:50.6354874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:50.6355579Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:50.6357886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:50.6358556Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:50.6451599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:50.6452119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:52.1763756Z ok (5.634s) 2022-03-17T19:21:52.1892243Z test_delayed_reduce_scatter_offload_true_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20162 2022-03-17T19:21:52.1998130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20163 2022-03-17T19:21:52.9919656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht32l3ri 2022-03-17T19:21:52.9920273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht32l3ri/_remote_module_non_sriptable.py 2022-03-17T19:21:53.0026986Z dist init r=1, world=2 2022-03-17T19:21:53.0031567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:53.0350323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4zezjma 2022-03-17T19:21:53.0352741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4zezjma/_remote_module_non_sriptable.py 2022-03-17T19:21:53.0455711Z dist init r=0, world=2 2022-03-17T19:21:53.0460054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:53.0461285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:53.0539487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:21:54.4134529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:21:54.4135273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:21:54.7286805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:54.7287333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:54.7405275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:54.7405827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:54.7538862Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:21:55.7600399Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:21:55.7601432Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T19:21:55.7621391Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:55.7622425Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:55.7623401Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:21:55.7624151Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:21:56.7995343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:56.7996681Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:56.7998450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:21:56.7999551Z warnings.warn(msg, FutureWarning) 2022-03-17T19:21:56.8091342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:56.8091842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:21:59.1122438Z ok (6.936s) 2022-03-17T19:21:59.1250466Z test_delayed_reduce_scatter_offload_true_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20249 2022-03-17T19:21:59.1354931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20250 2022-03-17T19:21:59.9301026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaakr754j 2022-03-17T19:21:59.9301876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaakr754j/_remote_module_non_sriptable.py 2022-03-17T19:21:59.9406904Z dist init r=1, world=2 2022-03-17T19:21:59.9411087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:21:59.9840009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpguc40mg8 2022-03-17T19:21:59.9843160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpguc40mg8/_remote_module_non_sriptable.py 2022-03-17T19:21:59.9947675Z dist init r=0, world=2 2022-03-17T19:21:59.9952031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:21:59.9952828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:00.0020479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:01.3567882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:01.3568443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:01.6717493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:22:01.6718010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:22:01.6835799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:22:01.6836299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:22:01.6961711Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:02.7020907Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:22:02.7022141Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T19:22:02.7042155Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:02.7042969Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:02.7043973Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:02.7044741Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:03.7407251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:03.7407959Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:03.7408918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:03.7409536Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:03.7502203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:22:03.7502749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:22:06.1479286Z ok (7.036s) 2022-03-17T19:22:06.1607990Z test_mixture_of_experts_offload_false_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20336 2022-03-17T19:22:06.1713177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20337 2022-03-17T19:22:07.0231695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbeom0xj 2022-03-17T19:22:07.0232545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbeom0xj/_remote_module_non_sriptable.py 2022-03-17T19:22:07.0259792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9511mzze 2022-03-17T19:22:07.0262618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9511mzze/_remote_module_non_sriptable.py 2022-03-17T19:22:07.0336952Z dist init r=0, world=2 2022-03-17T19:22:07.0341380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:07.0367306Z dist init r=1, world=2 2022-03-17T19:22:07.0371536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:07.0372600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:07.0444849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:08.3959297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:08.3959790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:08.7416919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:08.7420422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:08.7421714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:08.7518025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:08.7618621Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:08.7619403Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:08.7624838Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:08.7625613Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:08.7900847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:08.7901557Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:08.7904083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:08.7904742Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:08.8022926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:08.8031594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:08.8032261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:08.8126170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:09.1780751Z ok (3.030s) 2022-03-17T19:22:09.1911956Z test_mixture_of_experts_offload_false_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20427 2022-03-17T19:22:09.2018064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20428 2022-03-17T19:22:10.0044073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqagpzg9 2022-03-17T19:22:10.0045086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqagpzg9/_remote_module_non_sriptable.py 2022-03-17T19:22:10.0149979Z dist init r=1, world=2 2022-03-17T19:22:10.0154244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:10.0320683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppteqyf_g 2022-03-17T19:22:10.0323199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppteqyf_g/_remote_module_non_sriptable.py 2022-03-17T19:22:10.0426034Z dist init r=0, world=2 2022-03-17T19:22:10.0430283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:10.0431295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:10.0460083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:11.4023632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:11.4024143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:11.7153543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:11.7162022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:11.7162806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:11.7254280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:11.7354234Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:11.7355046Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:11.7359868Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:11.7360629Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:11.7634043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:11.7634951Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:11.7637467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:11.7638137Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:11.7756463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:11.7761165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:11.7762072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:11.7859178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:12.1085953Z ok (2.930s) 2022-03-17T19:22:12.1214397Z test_mixture_of_experts_offload_false_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20518 2022-03-17T19:22:12.1319682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20519 2022-03-17T19:22:12.9183303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwc1xrvp6 2022-03-17T19:22:12.9184643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwc1xrvp6/_remote_module_non_sriptable.py 2022-03-17T19:22:12.9288385Z dist init r=1, world=2 2022-03-17T19:22:12.9292966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:12.9761525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgk1aurt3 2022-03-17T19:22:12.9764429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgk1aurt3/_remote_module_non_sriptable.py 2022-03-17T19:22:12.9867965Z dist init r=0, world=2 2022-03-17T19:22:12.9872400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:12.9873466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:12.9901914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:14.3326260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:14.3326786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:14.6435679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:14.6438235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:14.6439353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:14.6536438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:14.6629914Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:14.6630725Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:14.6635225Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:14.6635988Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:14.6895726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:14.6896406Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:14.6897316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:14.6897970Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:14.7016833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:14.7019085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:14.7020002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:14.7119505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:15.0384534Z ok (2.930s) 2022-03-17T19:22:15.0513282Z test_mixture_of_experts_offload_false_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20609 2022-03-17T19:22:15.0618288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20610 2022-03-17T19:22:15.8627043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_4_ns94 2022-03-17T19:22:15.8628941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_4_ns94/_remote_module_non_sriptable.py 2022-03-17T19:22:15.8734543Z dist init r=1, world=2 2022-03-17T19:22:15.8738996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:15.9081547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpve4oljif 2022-03-17T19:22:15.9084204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpve4oljif/_remote_module_non_sriptable.py 2022-03-17T19:22:15.9188347Z dist init r=0, world=2 2022-03-17T19:22:15.9193024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:15.9193768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:15.9247196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:17.2861387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:17.2861937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:17.5969240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:17.6048431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:17.6049261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:17.6070714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:17.6166009Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:17.6166749Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:17.6171851Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:17.6172950Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:17.6440530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:17.6441171Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:17.6442855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:17.6443443Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:17.6566865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:17.6568153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:17.6569613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:17.6669771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:17.9682726Z ok (2.930s) 2022-03-17T19:22:17.9807943Z test_mixture_of_experts_offload_false_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20700 2022-03-17T19:22:17.9911896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20701 2022-03-17T19:22:18.7969386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoownuikt 2022-03-17T19:22:18.7970508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoownuikt/_remote_module_non_sriptable.py 2022-03-17T19:22:18.8075138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xyn03_b 2022-03-17T19:22:18.8076209Z dist init r=1, world=2 2022-03-17T19:22:18.8077480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xyn03_b/_remote_module_non_sriptable.py 2022-03-17T19:22:18.8080821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:18.8181575Z dist init r=0, world=2 2022-03-17T19:22:18.8186613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:18.8187363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:18.8285561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:20.1821419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:20.1821924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:20.4962738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:20.4992718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:20.4993473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:20.5064430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:20.5166416Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:20.5167196Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:20.5172199Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:20.5173219Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:20.5451803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:20.5452537Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:20.5453559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:20.5454225Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:20.5578008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:20.5582425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:20.5583137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:20.5680515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:20.8975496Z ok (2.929s) 2022-03-17T19:22:20.9100652Z test_mixture_of_experts_offload_false_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20791 2022-03-17T19:22:20.9204930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20792 2022-03-17T19:22:21.7350836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdw5sailm 2022-03-17T19:22:21.7351733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdw5sailm/_remote_module_non_sriptable.py 2022-03-17T19:22:21.7355636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1_4bru2 2022-03-17T19:22:21.7358636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1_4bru2/_remote_module_non_sriptable.py 2022-03-17T19:22:21.7456439Z dist init r=1, world=2 2022-03-17T19:22:21.7461010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:21.7463555Z dist init r=0, world=2 2022-03-17T19:22:21.7467916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:21.7468718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:21.7564185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:23.1274063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:23.1275050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:23.4405190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:23.4405737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:23.4406514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:23.4506135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:23.4606633Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:23.4607689Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:23.4612057Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:23.4612815Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:23.4884893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:23.4885601Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:23.4886513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:23.4887138Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:23.5010005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:23.5012454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:23.5013206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:23.5112490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:23.8268054Z ok (2.929s) 2022-03-17T19:22:23.8393239Z test_mixture_of_experts_offload_false_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20882 2022-03-17T19:22:23.8497211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20883 2022-03-17T19:22:24.6609539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppd6p_2_a 2022-03-17T19:22:24.6610369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppd6p_2_a/_remote_module_non_sriptable.py 2022-03-17T19:22:24.6716446Z dist init r=1, world=2 2022-03-17T19:22:24.6720927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:24.6846004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppimony55 2022-03-17T19:22:24.6848510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppimony55/_remote_module_non_sriptable.py 2022-03-17T19:22:24.6952152Z dist init r=0, world=2 2022-03-17T19:22:24.6956041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:24.6957149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:24.7026771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:26.0452373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:26.0453314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:26.3556004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:26.3566592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:26.3567638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:26.3657758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:26.3757815Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:26.3759600Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:26.3763742Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:26.3765310Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:26.4039083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:26.4040458Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:26.4042674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:26.4044037Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:26.4161780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:26.4172218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:26.4173827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:26.4263445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:26.7562217Z ok (2.929s) 2022-03-17T19:22:26.7692653Z test_mixture_of_experts_offload_false_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20973 2022-03-17T19:22:26.7797407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20974 2022-03-17T19:22:27.5792880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8h7tyrj 2022-03-17T19:22:27.5793416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8h7tyrj/_remote_module_non_sriptable.py 2022-03-17T19:22:27.5817776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgqx9v8g 2022-03-17T19:22:27.5820409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgqx9v8g/_remote_module_non_sriptable.py 2022-03-17T19:22:27.5898579Z dist init r=1, world=2 2022-03-17T19:22:27.5903326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:27.5924524Z dist init r=0, world=2 2022-03-17T19:22:27.5928815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:27.5929884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:27.6006962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:28.9496928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:28.9497619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:29.2661534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:29.2691036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:29.2692273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:29.2762796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:29.2862252Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:29.2863079Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:29.2866928Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:29.2867731Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:29.3138126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:29.3138824Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:29.3140945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:29.3141563Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:29.3264696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:29.3268360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:29.3269094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:29.3367365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:29.6862749Z ok (2.930s) 2022-03-17T19:22:29.6991588Z test_mixture_of_experts_offload_false_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21064 2022-03-17T19:22:29.7097486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21065 2022-03-17T19:22:30.5129585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4bw9n_v 2022-03-17T19:22:30.5130721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4bw9n_v/_remote_module_non_sriptable.py 2022-03-17T19:22:30.5236159Z dist init r=1, world=2 2022-03-17T19:22:30.5240498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:30.5504017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnmv55kw 2022-03-17T19:22:30.5506327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnmv55kw/_remote_module_non_sriptable.py 2022-03-17T19:22:30.5611228Z dist init r=0, world=2 2022-03-17T19:22:30.5616243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:30.5617586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:30.5647607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:31.9280317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:31.9281264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:32.2408438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:32.2409472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:32.2410900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:32.2412283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:32.2519252Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:32.2520772Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:32.2522794Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:32.2524253Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:32.2802198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:32.2803623Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:32.2806017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:32.2807266Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:32.2926070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:32.2931610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:32.2932647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:32.3028603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:32.6164320Z ok (2.930s) 2022-03-17T19:22:32.6292814Z test_mixture_of_experts_offload_false_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21155 2022-03-17T19:22:32.6397791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21156 2022-03-17T19:22:33.4433226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3_lf__2 2022-03-17T19:22:33.4433789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3a4dm249 2022-03-17T19:22:33.4434640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3_lf__2/_remote_module_non_sriptable.py 2022-03-17T19:22:33.4435197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3a4dm249/_remote_module_non_sriptable.py 2022-03-17T19:22:33.4540963Z dist init r=0, world=2 2022-03-17T19:22:33.4541488Z dist init r=1, world=2 2022-03-17T19:22:33.4546159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:33.4546939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:33.4547672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:33.4548367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:34.8380043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:34.8380945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:35.1533708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:35.1540341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:35.1541623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:35.1634695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:35.1737072Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:35.1737879Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:35.1742918Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:35.1743686Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:35.2018397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:35.2019087Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:35.2021569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:35.2022250Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:35.2140765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:35.2145700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:35.2146551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:35.2243529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:35.5462509Z ok (2.930s) 2022-03-17T19:22:35.5592251Z test_mixture_of_experts_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21246 2022-03-17T19:22:35.5697430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21247 2022-03-17T19:22:36.3661905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmetrd0wr 2022-03-17T19:22:36.3662754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmetrd0wr/_remote_module_non_sriptable.py 2022-03-17T19:22:36.3768002Z dist init r=1, world=2 2022-03-17T19:22:36.3772236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:36.4138226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75e7ytid 2022-03-17T19:22:36.4141008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75e7ytid/_remote_module_non_sriptable.py 2022-03-17T19:22:36.4244689Z dist init r=0, world=2 2022-03-17T19:22:36.4248944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:36.4250048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:36.4280233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:37.7804481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:37.7805126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:38.0952136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:38.0955443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:38.0956728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:38.1053963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:38.1152079Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:38.1152873Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:38.1157751Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:38.1158526Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:38.1432510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:38.1433169Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:38.1434076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:38.1434910Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:38.1558173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:38.1558691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:38.1559342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:38.1560195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:38.4764411Z ok (2.930s) 2022-03-17T19:22:38.4894726Z test_mixture_of_experts_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21337 2022-03-17T19:22:38.4999249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21338 2022-03-17T19:22:39.3003905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8qhtio6 2022-03-17T19:22:39.3005085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8qhtio6/_remote_module_non_sriptable.py 2022-03-17T19:22:39.3110228Z dist init r=1, world=2 2022-03-17T19:22:39.3114686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:39.3537557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc8dyum_ 2022-03-17T19:22:39.3539955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc8dyum_/_remote_module_non_sriptable.py 2022-03-17T19:22:39.3645123Z dist init r=0, world=2 2022-03-17T19:22:39.3649519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:39.3650963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:39.3724274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:40.7389482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:40.7389998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:41.0484589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:41.0576172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:41.0576971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:41.0585603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:41.0683207Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:41.0683997Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:41.0688359Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:41.0689128Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:41.0959671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:41.0960686Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:41.0961709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:41.0962372Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:41.1081664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:41.1087181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:41.1088102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:41.1184580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:41.5065201Z ok (3.030s) 2022-03-17T19:22:41.5191665Z test_mixture_of_experts_offload_true_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21428 2022-03-17T19:22:41.5295623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21429 2022-03-17T19:22:42.3258911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt0r0ahy 2022-03-17T19:22:42.3259856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt0r0ahy/_remote_module_non_sriptable.py 2022-03-17T19:22:42.3365806Z dist init r=1, world=2 2022-03-17T19:22:42.3370071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:42.3632019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvp6y_d61 2022-03-17T19:22:42.3634704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvp6y_d61/_remote_module_non_sriptable.py 2022-03-17T19:22:42.3738490Z dist init r=0, world=2 2022-03-17T19:22:42.3743211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:42.3744391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:42.3776932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:43.7583691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:43.7584511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:44.0682467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:44.0739633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:44.0740933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:44.0783582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:44.0870830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:44.0898586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:44.0899579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:44.0939777Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:44.0942562Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:44.0943828Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:44.0945087Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:44.0974420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:44.1015279Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:44.1016581Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:44.1017833Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:44.1019074Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:44.1150422Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:44.1151260Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:44.1152277Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:44.1153038Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:44.1497473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:44.1498311Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:44.1499249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:44.1499909Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:44.1616880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:22:44.1618245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:22:44.1619498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:44.1719127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:44.5364119Z ok (3.030s) 2022-03-17T19:22:44.5490852Z test_mixture_of_experts_offload_true_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21521 2022-03-17T19:22:44.5594661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21522 2022-03-17T19:22:45.3562756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lsmltan 2022-03-17T19:22:45.3563602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lsmltan/_remote_module_non_sriptable.py 2022-03-17T19:22:45.3606146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph21l_3zx 2022-03-17T19:22:45.3608898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph21l_3zx/_remote_module_non_sriptable.py 2022-03-17T19:22:45.3669088Z dist init r=0, world=2 2022-03-17T19:22:45.3673523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:45.3712388Z dist init r=1, world=2 2022-03-17T19:22:45.3716676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:45.3717468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:45.3776858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:46.7373778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:46.7374302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:47.0532205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:47.0561257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:47.0562045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:47.0633304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:47.0720996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:47.0749890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:47.0750578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:47.0790778Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:47.0792287Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:47.0793646Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:47.0794917Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:47.0824267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:47.0864609Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:47.0865901Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:47.0867141Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:47.0868376Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:47.0999405Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:47.1000161Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:47.1001972Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:47.1003154Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:47.1349180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:47.1350215Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:47.1351286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:47.1352192Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:47.1466813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:22:47.1471565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:22:47.1472523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:47.1569578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:47.4657880Z ok (2.929s) 2022-03-17T19:22:47.4784056Z test_mixture_of_experts_offload_true_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21614 2022-03-17T19:22:47.4888158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21615 2022-03-17T19:22:48.2747014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfzjkb1l 2022-03-17T19:22:48.2747812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfzjkb1l/_remote_module_non_sriptable.py 2022-03-17T19:22:48.2853344Z dist init r=1, world=2 2022-03-17T19:22:48.2857800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:48.2980379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporb2gtdc 2022-03-17T19:22:48.2983683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporb2gtdc/_remote_module_non_sriptable.py 2022-03-17T19:22:48.3089136Z dist init r=0, world=2 2022-03-17T19:22:48.3093585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:48.3094825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:48.3163754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:49.6924460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:49.6925060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:50.0085804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:50.0126772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:50.0127578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:50.0186766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:50.0274613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:50.0302401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:50.0303361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:50.0343158Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:50.0344720Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:50.0346069Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:50.0347333Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:50.0377942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:50.0418877Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:50.0420174Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:50.0421631Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:50.0422899Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:50.0549843Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:50.0550642Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:50.0553897Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:50.0554693Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:50.0896564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:50.0897278Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:50.0899199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:50.0900032Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:50.1015374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:22:50.1023521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:22:50.1024394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:50.1118163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:50.4958021Z ok (3.030s) 2022-03-17T19:22:50.5092828Z test_mixture_of_experts_offload_true_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21707 2022-03-17T19:22:50.5202650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21708 2022-03-17T19:22:51.3177039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkp5nwzr 2022-03-17T19:22:51.3177969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkp5nwzr/_remote_module_non_sriptable.py 2022-03-17T19:22:51.3282538Z dist init r=0, world=2 2022-03-17T19:22:51.3286783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:51.3607657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyark37xi 2022-03-17T19:22:51.3610267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyark37xi/_remote_module_non_sriptable.py 2022-03-17T19:22:51.3713680Z dist init r=1, world=2 2022-03-17T19:22:51.3718059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:51.3718847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:51.3794992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:52.7307432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:52.7307985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:53.0453827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:53.0466666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:53.0467579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:53.0554865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:53.0643873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:53.0673094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:53.0673957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:53.0714769Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:53.0716034Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:53.0717548Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:53.0718892Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:53.0747011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:53.0786726Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:53.0787991Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:53.0789215Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:53.0790468Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:53.0917029Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:53.0917787Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:53.0918797Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:53.0919568Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:53.1253915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:53.1254700Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:53.1255899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:53.1256723Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:53.1371921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:22:53.1377504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:22:53.1378384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:53.1474326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:53.5270520Z ok (3.031s) 2022-03-17T19:22:53.5398894Z test_mixture_of_experts_offload_true_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21800 2022-03-17T19:22:53.5504556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21801 2022-03-17T19:22:54.3477359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jezg81n 2022-03-17T19:22:54.3478205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jezg81n/_remote_module_non_sriptable.py 2022-03-17T19:22:54.3584727Z dist init r=1, world=2 2022-03-17T19:22:54.3589422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:54.3766426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2brnss72 2022-03-17T19:22:54.3769143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2brnss72/_remote_module_non_sriptable.py 2022-03-17T19:22:54.3872115Z dist init r=0, world=2 2022-03-17T19:22:54.3876366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:54.3877419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:54.3894819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:55.7319544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:55.7320109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:56.0455885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:56.0463969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:56.0464897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:56.0557140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:56.0645262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:56.0674838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:56.0676084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:56.0717312Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:56.0719753Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:56.0722474Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:56.0725079Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:56.0748252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:56.0789991Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:56.0793074Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:56.0796044Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:56.0799016Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:56.0928110Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:56.0929645Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:56.0931584Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:56.0933094Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:56.1274572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:56.1275925Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:56.1277669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:56.1278945Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:56.1395278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:22:56.1397853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:22:56.1398896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:56.1497833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:56.4569711Z ok (2.930s) 2022-03-17T19:22:56.4698119Z test_mixture_of_experts_offload_true_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21893 2022-03-17T19:22:56.4802836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21894 2022-03-17T19:22:57.3212324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0tni_bon 2022-03-17T19:22:57.3213451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0tni_bon/_remote_module_non_sriptable.py 2022-03-17T19:22:57.3266517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpns820zvx 2022-03-17T19:22:57.3269252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpns820zvx/_remote_module_non_sriptable.py 2022-03-17T19:22:57.3317877Z dist init r=0, world=2 2022-03-17T19:22:57.3322045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:22:57.3372882Z dist init r=1, world=2 2022-03-17T19:22:57.3377307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:22:57.3378535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:57.3424877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:22:58.7035703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:22:58.7036231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:22:59.0130378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:22:59.0181006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:22:59.0182033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:59.0231540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:22:59.0317623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:22:59.0345741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:22:59.0346549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:59.0385822Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:59.0387211Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:59.0388476Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:59.0389979Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:59.0420724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:22:59.0461463Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:59.0462720Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:59.0463977Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:59.0465218Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:22:59.0597173Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:59.0597948Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:59.0598956Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:22:59.0599726Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:22:59.0939990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:59.0940835Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:59.0941783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:22:59.0942414Z warnings.warn(msg, FutureWarning) 2022-03-17T19:22:59.1056939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:22:59.1062164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:22:59.1063322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:59.1159298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:22:59.4872863Z ok (3.030s) 2022-03-17T19:22:59.5003780Z test_mixture_of_experts_offload_true_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21986 2022-03-17T19:22:59.5111295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21987 2022-03-17T19:23:00.3358188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pdszak3 2022-03-17T19:23:00.3359195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pdszak3/_remote_module_non_sriptable.py 2022-03-17T19:23:00.3462057Z dist init r=0, world=2 2022-03-17T19:23:00.3466678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:00.3539732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kf5q27g 2022-03-17T19:23:00.3542349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kf5q27g/_remote_module_non_sriptable.py 2022-03-17T19:23:00.3646874Z dist init r=1, world=2 2022-03-17T19:23:00.3651325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:00.3652430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:00.3670874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:01.7485090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:01.7485639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:02.0625013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:02.0643082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:02.0643861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:02.0726307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:02.0812549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:02.0841300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:02.0842244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:02.0881897Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:02.0883182Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:02.0884449Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:02.0885942Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:02.0915404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:02.0955742Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:02.0957032Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:02.0958281Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:02.0959531Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:02.1089262Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:02.1090061Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:02.1091067Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:02.1091834Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:02.1429610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:02.1430317Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:02.1431592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:02.1432231Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:02.1548053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:02.1553638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:02.1554410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:02.1650613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:02.5179973Z ok (3.031s) 2022-03-17T19:23:02.5309987Z test_mixture_of_experts_offload_true_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22079 2022-03-17T19:23:02.5414638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22080 2022-03-17T19:23:03.3838501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1hu04fc 2022-03-17T19:23:03.3839126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1hu04fc/_remote_module_non_sriptable.py 2022-03-17T19:23:03.3927594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9y17nh8 2022-03-17T19:23:03.3930237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9y17nh8/_remote_module_non_sriptable.py 2022-03-17T19:23:03.3943264Z dist init r=0, world=2 2022-03-17T19:23:03.3947469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:03.4037829Z dist init r=1, world=2 2022-03-17T19:23:03.4042337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:03.4043414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:03.4050414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:04.7716631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:04.7717126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:05.1089053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:05.1089595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:05.1090309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:05.1090997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:05.1206220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:05.1206743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:05.1207348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:05.1208014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:05.1247416Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:05.1248715Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:05.1249969Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:05.1251489Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:05.1252833Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:05.1254083Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:05.1255328Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:05.1256537Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:05.1384061Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:05.1384821Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:05.1385847Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:05.1386592Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:05.1725587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:05.1726585Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:05.1727461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:05.1728087Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:05.1844807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:05.1848569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:05.1849244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:05.1947159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:05.5483729Z ok (3.030s) 2022-03-17T19:23:05.5612188Z test_mixture_of_experts_offload_true_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22172 2022-03-17T19:23:05.5718008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22173 2022-03-17T19:23:06.3643916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4a0zdrcm 2022-03-17T19:23:06.3644949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4a0zdrcm/_remote_module_non_sriptable.py 2022-03-17T19:23:06.3750547Z dist init r=1, world=2 2022-03-17T19:23:06.3754509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:06.4111473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fdcr0t6 2022-03-17T19:23:06.4114086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fdcr0t6/_remote_module_non_sriptable.py 2022-03-17T19:23:06.4216695Z dist init r=0, world=2 2022-03-17T19:23:06.4221003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:06.4222028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:06.4262970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:07.7881813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:07.7882315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:08.0999932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:08.1043497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:08.1044284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:08.1100877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:08.1186948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:08.1215512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:08.1216437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:08.1255823Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:08.1257119Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:08.1258365Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:08.1259622Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:08.1289798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:08.1330397Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:08.1331772Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:08.1333064Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:08.1334317Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:08.1467355Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:08.1468137Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:08.1469133Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:08.1469892Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:08.1812284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:08.1812990Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:08.1814575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:08.1815213Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:08.1930329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:08.1938146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:08.1938851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:08.2032728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:08.5786550Z ok (3.030s) 2022-03-17T19:23:08.5916128Z test_mixture_of_experts_offload_true_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22265 2022-03-17T19:23:08.6021201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22266 2022-03-17T19:23:09.4060498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwh_rpv8 2022-03-17T19:23:09.4061476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwh_rpv8/_remote_module_non_sriptable.py 2022-03-17T19:23:09.4166940Z dist init r=1, world=2 2022-03-17T19:23:09.4171424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:09.4364246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurd9jgu4 2022-03-17T19:23:09.4366901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurd9jgu4/_remote_module_non_sriptable.py 2022-03-17T19:23:09.4471310Z dist init r=0, world=2 2022-03-17T19:23:09.4475555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:09.4476364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:09.4477041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:10.8236504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:10.8236986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:11.1369184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:11.1390138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:11.1390883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:11.1469707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:11.1561296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:11.1590752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:11.1591641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:11.1632858Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:11.1634074Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:11.1635310Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:11.1636571Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:11.1663946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:11.1703935Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:11.1705309Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:11.1706585Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:11.1707848Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:11.1841734Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:11.1842563Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:11.1843544Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:11.1844313Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:11.2196398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:11.2197420Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:11.2198749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:11.2199383Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:11.2319876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:11.2322421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:11.2323144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:11.2421982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:11.6090120Z ok (3.030s) 2022-03-17T19:23:11.6218929Z test_mixture_of_experts_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22358 2022-03-17T19:23:11.6324875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22359 2022-03-17T19:23:12.4501002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20bjsryk 2022-03-17T19:23:12.4501844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20bjsryk/_remote_module_non_sriptable.py 2022-03-17T19:23:12.4607867Z dist init r=1, world=2 2022-03-17T19:23:12.4612175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:12.4839147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpps3suwbk 2022-03-17T19:23:12.4842302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpps3suwbk/_remote_module_non_sriptable.py 2022-03-17T19:23:12.4946572Z dist init r=0, world=2 2022-03-17T19:23:12.4950828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:12.4951963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:12.5019245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:13.8629669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:13.8630191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:14.2110491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:14.2157542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:14.2158298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:14.2211712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:14.2300530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:14.2329972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:14.2330613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:14.2372224Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:14.2373499Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:14.2374765Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:14.2376017Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:14.2403172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:14.2443285Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:14.2444557Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:14.2445910Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:14.2447171Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:14.2575745Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:14.2576552Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:14.2577477Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:14.2578240Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:14.2920155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:14.2920984Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:14.2922714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:14.2923272Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:14.3042172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:14.3045419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:14.3046100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:14.3144610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:14.6391457Z ok (3.030s) 2022-03-17T19:23:14.6518944Z test_mixture_of_experts_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22451 2022-03-17T19:23:14.6623044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22452 2022-03-17T19:23:15.4607569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyz2sdjr 2022-03-17T19:23:15.4608660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyz2sdjr/_remote_module_non_sriptable.py 2022-03-17T19:23:15.4715185Z dist init r=1, world=2 2022-03-17T19:23:15.4719599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:15.5128918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngjwi9et 2022-03-17T19:23:15.5131439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngjwi9et/_remote_module_non_sriptable.py 2022-03-17T19:23:15.5236278Z dist init r=0, world=2 2022-03-17T19:23:15.5240402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:15.5241627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:15.5329422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:16.8837831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:16.8838323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:17.1996627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:17.1997228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:17.1997966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:17.1998661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:17.2116822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:17.2117348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:17.2117959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:17.2118621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:17.2158412Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:17.2159663Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:17.2160915Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:17.2162140Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:17.2163387Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:17.2164883Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:17.2166208Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:17.2167460Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:17.2296848Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:17.2297598Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:17.2298625Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:17.2299391Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:17.2644109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:17.2644731Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:17.2645657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:17.2646312Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:17.2767251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:17.2768562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:17.2769498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:17.2869612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:17.6689581Z ok (3.030s) 2022-03-17T19:23:17.6817415Z test_mixture_of_experts_with_delay_before_free_offload_false_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22544 2022-03-17T19:23:17.6922815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22545 2022-03-17T19:23:18.5442859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhooh854 2022-03-17T19:23:18.5443473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppah17als 2022-03-17T19:23:18.5444274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhooh854/_remote_module_non_sriptable.py 2022-03-17T19:23:18.5444793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppah17als/_remote_module_non_sriptable.py 2022-03-17T19:23:18.5550310Z dist init r=1, world=2 2022-03-17T19:23:18.5550561Z dist init r=0, world=2 2022-03-17T19:23:18.5554907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:18.5555393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:18.5556294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:18.5556926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:19.9270721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:19.9271261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:20.2418599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:20.2461513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:20.2462243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:20.2519407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:20.5144051Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:20.5144792Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:20.5145812Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:20.5146573Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:20.7427506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:20.7428207Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:20.7429140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:20.7429680Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:20.7553687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:20.7556339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:20.7557023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:20.7656142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:21.3999515Z ok (3.731s) 2022-03-17T19:23:21.4126206Z test_mixture_of_experts_with_delay_before_free_offload_false_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22635 2022-03-17T19:23:21.4230323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22636 2022-03-17T19:23:22.2217152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuv6mdsi 2022-03-17T19:23:22.2218092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuv6mdsi/_remote_module_non_sriptable.py 2022-03-17T19:23:22.2324412Z dist init r=0, world=2 2022-03-17T19:23:22.2328828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:22.2647177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu02v00ok 2022-03-17T19:23:22.2649831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu02v00ok/_remote_module_non_sriptable.py 2022-03-17T19:23:22.2753293Z dist init r=1, world=2 2022-03-17T19:23:22.2757495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:22.2758776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:22.2837013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:23.6432573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:23.6433101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:23.9613682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:23.9633048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:23.9633833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:23.9715455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:23.9810955Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:23.9811657Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:23.9816283Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:23.9816949Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:24.0088603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:24.0089282Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:24.0090109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:24.0091030Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:24.0215162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:24.0215673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:24.0216330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:24.0216914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:24.4299013Z ok (3.030s) 2022-03-17T19:23:24.4429366Z test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22726 2022-03-17T19:23:24.4534269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22727 2022-03-17T19:23:25.2491067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40b9iu4g 2022-03-17T19:23:25.2492085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40b9iu4g/_remote_module_non_sriptable.py 2022-03-17T19:23:25.2596891Z dist init r=1, world=2 2022-03-17T19:23:25.2601149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:25.3045261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xxca4le 2022-03-17T19:23:25.3047668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xxca4le/_remote_module_non_sriptable.py 2022-03-17T19:23:25.3151161Z dist init r=0, world=2 2022-03-17T19:23:25.3155688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:25.3156814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:25.3210593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:26.6832841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:26.6833743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:26.9962485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:26.9989662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:26.9991212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:27.0064294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:27.2683900Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:27.2684711Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:27.2685740Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:27.2686724Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:27.4971218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:27.4972385Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:27.4973968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:27.4974633Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:27.5094627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:27.5101341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:27.5102467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:27.5197276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:28.1610227Z ok (3.731s) 2022-03-17T19:23:28.1739238Z test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22817 2022-03-17T19:23:28.1843610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22818 2022-03-17T19:23:28.9861872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtzrh58f 2022-03-17T19:23:28.9863401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtzrh58f/_remote_module_non_sriptable.py 2022-03-17T19:23:28.9968050Z dist init r=1, world=2 2022-03-17T19:23:28.9972792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:29.0207891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp827shkd1 2022-03-17T19:23:29.0210514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp827shkd1/_remote_module_non_sriptable.py 2022-03-17T19:23:29.0313590Z dist init r=0, world=2 2022-03-17T19:23:29.0318105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:29.0319629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:29.0379906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:30.3966970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:30.3967486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:30.7094585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:30.7095138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:30.7095889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:30.7195268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:30.7293238Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:30.7294014Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:30.7299176Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:30.7300175Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:30.7569512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:30.7570431Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:30.7572071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:30.7572694Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:30.7691868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:30.7700427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:30.7701332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:30.7794630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:31.0908780Z ok (2.930s) 2022-03-17T19:23:31.1034636Z test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22908 2022-03-17T19:23:31.1139088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22909 2022-03-17T19:23:31.9016016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphofetcm8 2022-03-17T19:23:31.9017047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphofetcm8/_remote_module_non_sriptable.py 2022-03-17T19:23:31.9123479Z dist init r=1, world=2 2022-03-17T19:23:31.9127899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:31.9367263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hm7rkln 2022-03-17T19:23:31.9369692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hm7rkln/_remote_module_non_sriptable.py 2022-03-17T19:23:31.9472812Z dist init r=0, world=2 2022-03-17T19:23:31.9477048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:31.9478112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:31.9534955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:33.2987855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:33.2988348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:33.6144441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:33.6169902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:33.6170879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:33.6245673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:33.8872554Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:33.8873276Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:33.8874451Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:33.8875227Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:34.1588559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:34.1589181Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:34.1590462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:34.1591047Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:34.1708791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:34.1716016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:34.1716740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:34.1811213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:35.0218886Z ok (3.931s) 2022-03-17T19:23:35.0349616Z test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22999 2022-03-17T19:23:35.0455898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23000 2022-03-17T19:23:35.8409015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g45mpql 2022-03-17T19:23:35.8410129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g45mpql/_remote_module_non_sriptable.py 2022-03-17T19:23:35.8509030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3_a0vpo 2022-03-17T19:23:35.8511621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3_a0vpo/_remote_module_non_sriptable.py 2022-03-17T19:23:35.8515774Z dist init r=1, world=2 2022-03-17T19:23:35.8520382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:35.8631283Z dist init r=0, world=2 2022-03-17T19:23:35.8635664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:35.8636789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:35.8725025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:37.2279857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:37.2280380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:37.5371923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:37.5421671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:37.5422531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:37.5472595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:37.5569264Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:37.5570072Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:37.5574692Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:37.5575415Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:37.5841716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:37.5842367Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:37.5843274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:37.5843932Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:37.5964251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:37.5964768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:37.5965384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:37.5966072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:37.9521018Z ok (2.930s) 2022-03-17T19:23:37.9664722Z test_mixture_of_experts_with_delay_before_free_offload_true_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23090 2022-03-17T19:23:37.9756139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23091 2022-03-17T19:23:38.8395537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplljwl3lq 2022-03-17T19:23:38.8396803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplljwl3lq/_remote_module_non_sriptable.py 2022-03-17T19:23:38.8408834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmar_yfgr 2022-03-17T19:23:38.8411474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmar_yfgr/_remote_module_non_sriptable.py 2022-03-17T19:23:38.8501480Z dist init r=1, world=2 2022-03-17T19:23:38.8506528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:38.8516853Z dist init r=0, world=2 2022-03-17T19:23:38.8521097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:38.8522050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:38.8609900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:40.2151803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:40.2152290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:40.5283115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:40.5325954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:40.5326732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:40.5384121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:40.5472272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:40.5501506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:40.5502436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:40.5544534Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:40.5545838Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:40.5547099Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:40.5548353Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:40.5574597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:40.5623643Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:40.5624929Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:40.5626189Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:40.5627640Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:40.8313282Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:40.8314154Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:40.8315197Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:40.8315899Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:41.1130698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:41.1131465Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:41.1133845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:41.1134507Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:41.1250377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:41.1257021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:41.1257672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:41.1352943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:41.9836515Z ok (4.031s) 2022-03-17T19:23:41.9966136Z test_mixture_of_experts_with_delay_before_free_offload_true_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23183 2022-03-17T19:23:42.0070813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23184 2022-03-17T19:23:42.8050801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeegnl6bb 2022-03-17T19:23:42.8052024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeegnl6bb/_remote_module_non_sriptable.py 2022-03-17T19:23:42.8157701Z dist init r=1, world=2 2022-03-17T19:23:42.8162240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:42.8558734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbh28f22g 2022-03-17T19:23:42.8561390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbh28f22g/_remote_module_non_sriptable.py 2022-03-17T19:23:42.8663639Z dist init r=0, world=2 2022-03-17T19:23:42.8667765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:42.8668908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:42.8670164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:44.2201116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:44.2201637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:44.5331003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:44.5331577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:44.5332604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:44.5333303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:44.5444863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:44.5448237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:44.5449167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:44.5490199Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:44.5491959Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:44.5493524Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:44.5495006Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:44.5547176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:44.5595067Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:44.5596343Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:44.5597602Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:44.5598832Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:44.5717260Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:44.5718090Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:44.5719539Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:44.5720471Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:44.6053943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:44.6054631Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:44.6055543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:44.6056199Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:44.6171368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:44.6171995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:44.6172895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:44.6174171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:45.0136607Z ok (3.030s) 2022-03-17T19:23:45.0266575Z test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23276 2022-03-17T19:23:45.0371834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23277 2022-03-17T19:23:45.8277962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugj57roj 2022-03-17T19:23:45.8279441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugj57roj/_remote_module_non_sriptable.py 2022-03-17T19:23:45.8386283Z dist init r=1, world=2 2022-03-17T19:23:45.8390976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:45.8441703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s13_etj 2022-03-17T19:23:45.8444426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s13_etj/_remote_module_non_sriptable.py 2022-03-17T19:23:45.8547997Z dist init r=0, world=2 2022-03-17T19:23:45.8552220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:45.8552968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:45.8595358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:47.2361029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:47.2361546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:47.5553975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:47.5554516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:47.5555284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:47.5556189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:47.5674293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:47.5674960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:47.5675641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:47.5676303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:47.5716779Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:47.5718361Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:47.5719885Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:47.5721404Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:47.5725289Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:47.5726837Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:47.5728525Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:47.5730053Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:47.8423833Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:47.8424877Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:47.8426388Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:47.8427415Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:48.1254122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:48.1254830Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:48.1255768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:48.1256434Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:48.1378539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:48.1379291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:48.1380013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:48.1381127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:48.9452184Z ok (3.931s) 2022-03-17T19:23:48.9581719Z test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23369 2022-03-17T19:23:48.9687353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23370 2022-03-17T19:23:49.7916494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpthz1smio 2022-03-17T19:23:49.7917580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpthz1smio/_remote_module_non_sriptable.py 2022-03-17T19:23:49.7924808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_7dz1dk 2022-03-17T19:23:49.7927531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_7dz1dk/_remote_module_non_sriptable.py 2022-03-17T19:23:49.8023961Z dist init r=0, world=2 2022-03-17T19:23:49.8028644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:49.8033705Z dist init r=1, world=2 2022-03-17T19:23:49.8038285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:49.8039520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:49.8132138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:51.2042816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:51.2043710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:51.5160179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:51.5198797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:51.5199596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:51.5261119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:51.5349284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:51.5378104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:51.5378760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:51.5420532Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:51.5421854Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:51.5423112Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:51.5424369Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:51.5451619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:51.5499602Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:51.5501076Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:51.5502343Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:51.5503614Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:51.5624364Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:51.5625478Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:51.5626598Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:51.5627343Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:51.5966890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:51.5967565Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:51.5968489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:51.5969166Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:51.6088861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:51.6090211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:51.6091172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:51.6191228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:51.9754292Z ok (3.030s) 2022-03-17T19:23:51.9881584Z test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23462 2022-03-17T19:23:51.9987526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23463 2022-03-17T19:23:52.7997543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpef5e91 2022-03-17T19:23:52.7998508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpef5e91/_remote_module_non_sriptable.py 2022-03-17T19:23:52.8105017Z dist init r=1, world=2 2022-03-17T19:23:52.8109961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:52.8330643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsqi402p 2022-03-17T19:23:52.8333182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsqi402p/_remote_module_non_sriptable.py 2022-03-17T19:23:52.8437242Z dist init r=0, world=2 2022-03-17T19:23:52.8441657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:52.8442442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:52.8517279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:54.2174851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:54.2175398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:54.5327147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:54.5337238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:54.5338027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:54.5428147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:54.5518858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:54.5550042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:54.5550750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:54.5592500Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:54.5593767Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:54.5595019Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:54.5596284Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:54.5621505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:54.5669373Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:54.5670650Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:54.5671901Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:54.5673150Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:54.8358689Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:54.8359686Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:54.8360720Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:54.8361560Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:55.1178935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:55.1179959Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:55.1181902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:55.1182747Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:55.1297236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:55.1306556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:55.1307194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:55.1399630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:56.0069704Z ok (4.031s) 2022-03-17T19:23:56.0198111Z test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23555 2022-03-17T19:23:56.0302790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23556 2022-03-17T19:23:56.8254795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2aof4o3 2022-03-17T19:23:56.8256067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2aof4o3/_remote_module_non_sriptable.py 2022-03-17T19:23:56.8364852Z dist init r=1, world=2 2022-03-17T19:23:56.8369213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:56.8657132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxf1qmzg 2022-03-17T19:23:56.8659650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxf1qmzg/_remote_module_non_sriptable.py 2022-03-17T19:23:56.8764466Z dist init r=0, world=2 2022-03-17T19:23:56.8768905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:56.8770140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:56.8775844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:58.2422799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:23:58.2423495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:23:58.5582531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:23:58.5586471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:23:58.5587765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:58.5683759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:23:58.5770542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T19:23:58.5799965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T19:23:58.5801038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:58.5848833Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:58.5850397Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:58.5851920Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:58.5853441Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:58.5873417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T19:23:58.5914757Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:58.5916270Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:58.5917588Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:58.5918849Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:23:58.6050390Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:58.6051316Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:58.6052341Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:23:58.6053106Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:23:58.6396223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:58.6396906Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:58.6399910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:23:58.6400732Z warnings.warn(msg, FutureWarning) 2022-03-17T19:23:58.6515164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T19:23:58.6523430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T19:23:58.6524388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:58.6617670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T19:23:59.0371408Z ok (3.030s) 2022-03-17T19:23:59.0499210Z test_nested_all_wrapped_model_offload_false_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23648 2022-03-17T19:23:59.0605261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23649 2022-03-17T19:23:59.8610461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwaz7o5fv 2022-03-17T19:23:59.8611703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwaz7o5fv/_remote_module_non_sriptable.py 2022-03-17T19:23:59.8657011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f7qqufv 2022-03-17T19:23:59.8659674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f7qqufv/_remote_module_non_sriptable.py 2022-03-17T19:23:59.8718965Z dist init r=1, world=2 2022-03-17T19:23:59.8723464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:23:59.8764154Z dist init r=0, world=2 2022-03-17T19:23:59.8768675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:23:59.8769463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:23:59.8826980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:01.2466070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:01.2466592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:01.5586135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:01.5586618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:01.5758880Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:01.5759997Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:01.5761029Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:01.5761885Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:01.6200691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:01.6201388Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:01.6203504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:01.6204156Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:01.6281174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:01.6281674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:01.9670810Z ok (2.930s) 2022-03-17T19:24:01.9801047Z test_nested_all_wrapped_model_offload_false_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23735 2022-03-17T19:24:01.9906858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23736 2022-03-17T19:24:02.8153918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rwp0y7a 2022-03-17T19:24:02.8154941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rwp0y7a/_remote_module_non_sriptable.py 2022-03-17T19:24:02.8259999Z dist init r=0, world=2 2022-03-17T19:24:02.8265145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:02.8349650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjoewvfbr 2022-03-17T19:24:02.8352130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjoewvfbr/_remote_module_non_sriptable.py 2022-03-17T19:24:02.8455858Z dist init r=1, world=2 2022-03-17T19:24:02.8459919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:02.8460988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:02.8468856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:04.2213714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:04.2214195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:04.5340973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:04.5349456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:04.5518512Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:04.5519579Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:04.5526078Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:04.5526875Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:04.5969151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:04.5970281Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:04.5971484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:04.5972125Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:04.6050442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:04.6050940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:04.9977331Z ok (3.031s) 2022-03-17T19:24:05.0106568Z test_nested_all_wrapped_model_offload_false_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23822 2022-03-17T19:24:05.0213521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23823 2022-03-17T19:24:05.8253334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplboegbt6 2022-03-17T19:24:05.8255773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplboegbt6/_remote_module_non_sriptable.py 2022-03-17T19:24:05.8361492Z dist init r=1, world=2 2022-03-17T19:24:05.8366228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:05.8372579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty5f20ms 2022-03-17T19:24:05.8375915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty5f20ms/_remote_module_non_sriptable.py 2022-03-17T19:24:05.8481215Z dist init r=0, world=2 2022-03-17T19:24:05.8485705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:05.8487050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:05.8570691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:07.2390583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:07.2391039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:07.5518197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:07.5518746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:07.5679357Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:07.5680379Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:07.5681401Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:07.5682174Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:07.6098662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:07.6099354Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:07.6100470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:07.6101147Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:07.6179880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:07.6180532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:07.9279644Z ok (2.930s) 2022-03-17T19:24:07.9409841Z test_nested_all_wrapped_model_offload_false_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23909 2022-03-17T19:24:07.9516082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23910 2022-03-17T19:24:08.7797244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tqo1phl 2022-03-17T19:24:08.7798535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tqo1phl/_remote_module_non_sriptable.py 2022-03-17T19:24:08.7829560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90lgj8u_ 2022-03-17T19:24:08.7832564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90lgj8u_/_remote_module_non_sriptable.py 2022-03-17T19:24:08.7903730Z dist init r=0, world=2 2022-03-17T19:24:08.7908230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:08.7938037Z dist init r=1, world=2 2022-03-17T19:24:08.7943377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:08.7944437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:08.8012088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:10.1754812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:10.1755395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:10.4832358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:10.4835113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:10.4987555Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:10.4988317Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:10.4990439Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:10.4991096Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:10.5401698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:10.5402424Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:10.5403399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:10.5404081Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:10.5481120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:10.5481615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:10.8581918Z ok (2.930s) 2022-03-17T19:24:10.8711083Z test_nested_all_wrapped_model_offload_false_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23996 2022-03-17T19:24:10.8817184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23997 2022-03-17T19:24:11.6716022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4_3akm4 2022-03-17T19:24:11.6716988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4_3akm4/_remote_module_non_sriptable.py 2022-03-17T19:24:11.6777983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqnr6dap 2022-03-17T19:24:11.6780002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqnr6dap/_remote_module_non_sriptable.py 2022-03-17T19:24:11.6823646Z dist init r=0, world=2 2022-03-17T19:24:11.6828215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:11.6885611Z dist init r=1, world=2 2022-03-17T19:24:11.6890011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:11.6890812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:11.6931659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:13.0800025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:13.0800556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:13.3899466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:13.3908570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:13.4076466Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:13.4077286Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:13.4083012Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:13.4083797Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:13.4516817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:13.4517623Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:13.4518839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:13.4519520Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:13.4597380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:13.4597885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:13.7882748Z ok (2.930s) 2022-03-17T19:24:13.8011521Z test_nested_all_wrapped_model_offload_false_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24083 2022-03-17T19:24:13.8118200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24084 2022-03-17T19:24:14.6157536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3x_3lz6 2022-03-17T19:24:14.6158366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3x_3lz6/_remote_module_non_sriptable.py 2022-03-17T19:24:14.6264894Z dist init r=1, world=2 2022-03-17T19:24:14.6269146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:14.6274279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpndarnom1 2022-03-17T19:24:14.6277678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpndarnom1/_remote_module_non_sriptable.py 2022-03-17T19:24:14.6382483Z dist init r=0, world=2 2022-03-17T19:24:14.6387345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:14.6388233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:14.6473840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:16.0155247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:16.0155770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:16.3263819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:16.3271570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:16.3440725Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:16.3441474Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:16.3447165Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:16.3879076Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:16.3880061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:16.3880914Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:16.3883136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:16.3883772Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:16.3962000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:16.3962460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:16.8184530Z ok (3.030s) 2022-03-17T19:24:16.8313416Z test_nested_all_wrapped_model_offload_false_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24170 2022-03-17T19:24:16.8419542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24171 2022-03-17T19:24:17.6623052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vd6xqeq 2022-03-17T19:24:17.6624103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vd6xqeq/_remote_module_non_sriptable.py 2022-03-17T19:24:17.6730061Z dist init r=0, world=2 2022-03-17T19:24:17.6734981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:17.7055938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3lp0854 2022-03-17T19:24:17.7058254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3lp0854/_remote_module_non_sriptable.py 2022-03-17T19:24:17.7162297Z dist init r=1, world=2 2022-03-17T19:24:17.7166647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:17.7167341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:17.7243173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:19.1009322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:19.1009851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:19.4120598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:19.4121130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:19.4286389Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:19.4287207Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:19.4291835Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:19.4292800Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:19.4712160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:19.4712877Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:19.4715139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:19.4715809Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:19.4797140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:19.4797635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:19.8489049Z ok (3.030s) 2022-03-17T19:24:19.8625772Z test_nested_all_wrapped_model_offload_false_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24257 2022-03-17T19:24:19.8736533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24258 2022-03-17T19:24:20.7138557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5mnxe8h 2022-03-17T19:24:20.7139399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5mnxe8h/_remote_module_non_sriptable.py 2022-03-17T19:24:20.7157576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1q20kabv 2022-03-17T19:24:20.7160283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1q20kabv/_remote_module_non_sriptable.py 2022-03-17T19:24:20.7246243Z dist init r=0, world=2 2022-03-17T19:24:20.7250722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:20.7266182Z dist init r=1, world=2 2022-03-17T19:24:20.7270239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:20.7271134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:20.7354153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:22.1169100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:22.1169609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:22.4242692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:22.4250099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:22.4404522Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:22.4405318Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:22.4411026Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:22.4412019Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:22.4829296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:22.4829976Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:22.4831705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:22.4832316Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:22.4909856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:22.4910393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:22.8803426Z ok (3.031s) 2022-03-17T19:24:22.8932150Z test_nested_all_wrapped_model_offload_false_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24344 2022-03-17T19:24:22.9038584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24345 2022-03-17T19:24:23.7239414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcg0ofl2s 2022-03-17T19:24:23.7240513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcg0ofl2s/_remote_module_non_sriptable.py 2022-03-17T19:24:23.7250236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8_woph6 2022-03-17T19:24:23.7252984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8_woph6/_remote_module_non_sriptable.py 2022-03-17T19:24:23.7346153Z dist init r=0, world=2 2022-03-17T19:24:23.7350597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:23.7366633Z dist init r=1, world=2 2022-03-17T19:24:23.7371678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:23.7373027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:23.7454356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:25.1087359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:25.1088347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:25.4217259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:25.4218067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:25.4400984Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:25.4402511Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:25.4404580Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:25.4406522Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:25.4842430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:25.4843806Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:25.4845892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:25.4847155Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:25.4924779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:25.4925300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:25.8103762Z ok (2.930s) 2022-03-17T19:24:25.8231827Z test_nested_all_wrapped_model_offload_false_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24431 2022-03-17T19:24:25.8335937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24432 2022-03-17T19:24:26.6354290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbnpqj9z 2022-03-17T19:24:26.6355369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbnpqj9z/_remote_module_non_sriptable.py 2022-03-17T19:24:26.6460819Z dist init r=1, world=2 2022-03-17T19:24:26.6465361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:26.6679087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92bl0b78 2022-03-17T19:24:26.6681754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92bl0b78/_remote_module_non_sriptable.py 2022-03-17T19:24:26.6784046Z dist init r=0, world=2 2022-03-17T19:24:26.6788276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:26.6789199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:26.6872634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:28.0471671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:28.0472190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:28.3585093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:28.3594245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:28.3765733Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:28.3766520Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:28.3772875Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:28.3773659Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:28.4213042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:28.4213701Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:28.4218947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:28.4219679Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:28.4296878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:28.4297899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:28.7402055Z ok (2.930s) 2022-03-17T19:24:28.7530271Z test_nested_all_wrapped_model_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24518 2022-03-17T19:24:28.7635477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24519 2022-03-17T19:24:29.5870791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_2bi476 2022-03-17T19:24:29.5872190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_2bi476/_remote_module_non_sriptable.py 2022-03-17T19:24:29.5978279Z dist init r=1, world=2 2022-03-17T19:24:29.5983461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:29.6022043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnufqiph6 2022-03-17T19:24:29.6025305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnufqiph6/_remote_module_non_sriptable.py 2022-03-17T19:24:29.6128603Z dist init r=0, world=2 2022-03-17T19:24:29.6133413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:29.6134344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:29.6188152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:30.9772789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:30.9773340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:31.2867777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:31.2876211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:31.3031866Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:31.3032643Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:31.3038189Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:31.3038923Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:31.3460081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:31.3461303Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:31.3465306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:31.3466288Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:31.3542962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:31.3544026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:31.6701608Z ok (2.930s) 2022-03-17T19:24:31.6831546Z test_nested_all_wrapped_model_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24605 2022-03-17T19:24:31.6934721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24606 2022-03-17T19:24:32.5167095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8x4jq4ft 2022-03-17T19:24:32.5168365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8x4jq4ft/_remote_module_non_sriptable.py 2022-03-17T19:24:32.5274562Z dist init r=1, world=2 2022-03-17T19:24:32.5278821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:32.5379262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcronv5q 2022-03-17T19:24:32.5382119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcronv5q/_remote_module_non_sriptable.py 2022-03-17T19:24:32.5486754Z dist init r=0, world=2 2022-03-17T19:24:32.5491549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:32.5492607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:32.5585469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:33.9237032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:33.9237563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:34.2353262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:34.2353778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:34.2520957Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:34.2521782Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:34.2522812Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:34.2523564Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:34.2944252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:34.2945434Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:34.2946464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:34.2947122Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:34.3023907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:34.3024400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:34.7001155Z ok (3.030s) 2022-03-17T19:24:34.7135029Z test_nested_all_wrapped_model_offload_true_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24692 2022-03-17T19:24:34.7243885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24693 2022-03-17T19:24:35.5329872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3ykgx2x 2022-03-17T19:24:35.5330984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3ykgx2x/_remote_module_non_sriptable.py 2022-03-17T19:24:35.5436702Z dist init r=0, world=2 2022-03-17T19:24:35.5441167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:35.5447684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlq__2v3 2022-03-17T19:24:35.5450422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlq__2v3/_remote_module_non_sriptable.py 2022-03-17T19:24:35.5553913Z dist init r=1, world=2 2022-03-17T19:24:35.5557887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:35.5558755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:35.5645757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:36.9318800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:36.9319317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:37.2485541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:37.2486032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:37.2595411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:37.2595814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:37.2818117Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:37.2818927Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:37.2819957Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:37.2821004Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:37.3316395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:37.3317068Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:37.3319867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:37.3320652Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:37.3396958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:37.3397418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:37.7312648Z ok (3.031s) 2022-03-17T19:24:37.7442076Z test_nested_all_wrapped_model_offload_true_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24779 2022-03-17T19:24:37.7548267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24780 2022-03-17T19:24:38.5715834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc111g4dl 2022-03-17T19:24:38.5717018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc111g4dl/_remote_module_non_sriptable.py 2022-03-17T19:24:38.5822064Z dist init r=0, world=2 2022-03-17T19:24:38.5826844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:38.6052756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4djl2a_8 2022-03-17T19:24:38.6055633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4djl2a_8/_remote_module_non_sriptable.py 2022-03-17T19:24:38.6161054Z dist init r=1, world=2 2022-03-17T19:24:38.6165931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:38.6166937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:38.6234056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:39.9962346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:39.9963275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:40.3057495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:40.3062344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:40.3172728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:40.3174212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:40.3392171Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:40.3393781Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:40.3395835Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:40.3397803Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:40.3878349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:40.3879677Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:40.3881599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:40.3882880Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:40.3959794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:40.3961183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:40.7616910Z ok (3.030s) 2022-03-17T19:24:40.7744401Z test_nested_all_wrapped_model_offload_true_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24866 2022-03-17T19:24:40.7848105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24867 2022-03-17T19:24:41.5883536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuh4b5qwm 2022-03-17T19:24:41.5884185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuh4b5qwm/_remote_module_non_sriptable.py 2022-03-17T19:24:41.5987474Z dist init r=0, world=2 2022-03-17T19:24:41.5991663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:41.6016622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vde61j3 2022-03-17T19:24:41.6019493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vde61j3/_remote_module_non_sriptable.py 2022-03-17T19:24:41.6124458Z dist init r=1, world=2 2022-03-17T19:24:41.6128679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:41.6129821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:41.6196108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:42.9715799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:42.9716321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:43.2813316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:43.2822233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:43.2931986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:43.2934062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:43.3136542Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:43.3137315Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:43.3138321Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:43.3139359Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:43.3612614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:43.3613333Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:43.3616938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:43.3617623Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:43.3693539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:43.3695549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:43.6912658Z ok (2.929s) 2022-03-17T19:24:43.7042462Z test_nested_all_wrapped_model_offload_true_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24953 2022-03-17T19:24:43.7147010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24954 2022-03-17T19:24:44.5598309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9i8fx4j 2022-03-17T19:24:44.5599270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9i8fx4j/_remote_module_non_sriptable.py 2022-03-17T19:24:44.5704389Z dist init r=1, world=2 2022-03-17T19:24:44.5709156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:44.5774926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pjtrs9x 2022-03-17T19:24:44.5777997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pjtrs9x/_remote_module_non_sriptable.py 2022-03-17T19:24:44.5882584Z dist init r=0, world=2 2022-03-17T19:24:44.5886783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:44.5887731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:44.5913333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:45.9857665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:45.9858173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:46.3015456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:46.3015961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:46.3127050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:46.3127520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:46.3331808Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:46.3332538Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:46.3333553Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:46.3334606Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:46.3804852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:46.3805547Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:46.3807986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:46.3808650Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:46.3886815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:46.3887246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:46.7214008Z ok (3.030s) 2022-03-17T19:24:46.7342414Z test_nested_all_wrapped_model_offload_true_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25040 2022-03-17T19:24:46.7446959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25041 2022-03-17T19:24:47.5734195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dq6wncs 2022-03-17T19:24:47.5735170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dq6wncs/_remote_module_non_sriptable.py 2022-03-17T19:24:47.5841253Z dist init r=0, world=2 2022-03-17T19:24:47.5845522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:47.5850858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzbci4nl 2022-03-17T19:24:47.5854014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzbci4nl/_remote_module_non_sriptable.py 2022-03-17T19:24:47.5959377Z dist init r=1, world=2 2022-03-17T19:24:47.5963901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:47.5964671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:47.6050332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:48.9781153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:48.9781711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:49.2893992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:49.2894571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:49.3004735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:49.3005211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:49.3239483Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:49.3240266Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:49.3241595Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:49.3242368Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:49.3746706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:49.3747418Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:49.3749596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:49.3750286Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:49.3833639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:49.3836579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:49.7516452Z ok (3.030s) 2022-03-17T19:24:49.7644491Z test_nested_all_wrapped_model_offload_true_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25127 2022-03-17T19:24:49.7748962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25128 2022-03-17T19:24:50.5947065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ttjsio8 2022-03-17T19:24:50.5947698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ttjsio8/_remote_module_non_sriptable.py 2022-03-17T19:24:50.6050754Z dist init r=1, world=2 2022-03-17T19:24:50.6054959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:50.6063017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo71ikg5c 2022-03-17T19:24:50.6066513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo71ikg5c/_remote_module_non_sriptable.py 2022-03-17T19:24:50.6170231Z dist init r=0, world=2 2022-03-17T19:24:50.6174932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:50.6176078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:50.6259240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:51.9773792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:51.9774319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:52.2938149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:52.2938651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:52.3046943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:52.3047645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:52.3274396Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:52.3275496Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:52.3276528Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:52.3277286Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:52.3768657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:52.3769553Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:52.3770540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:52.3771441Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:52.3847920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:52.3848392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:52.7815498Z ok (3.030s) 2022-03-17T19:24:52.7946362Z test_nested_all_wrapped_model_offload_true_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25214 2022-03-17T19:24:52.8052014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25215 2022-03-17T19:24:53.6572166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66poaipc 2022-03-17T19:24:53.6573016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66poaipc/_remote_module_non_sriptable.py 2022-03-17T19:24:53.6592522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxzggrdq 2022-03-17T19:24:53.6595519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxzggrdq/_remote_module_non_sriptable.py 2022-03-17T19:24:53.6679016Z dist init r=0, world=2 2022-03-17T19:24:53.6683345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:53.6701423Z dist init r=1, world=2 2022-03-17T19:24:53.6706340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:53.6707429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:53.6786853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:55.0486455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:55.0487080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:55.3933802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:55.3934385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:55.4043814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:55.4044313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:55.4263360Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:55.4264515Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:55.4265531Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:55.4266400Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:55.4752665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:55.4753379Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:55.4756095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:55.4756777Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:55.4833475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:55.4833978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:55.8118993Z ok (3.030s) 2022-03-17T19:24:55.8250908Z test_nested_all_wrapped_model_offload_true_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25301 2022-03-17T19:24:55.8355214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25302 2022-03-17T19:24:56.6292757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73qxcyol 2022-03-17T19:24:56.6293837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73qxcyol/_remote_module_non_sriptable.py 2022-03-17T19:24:56.6398999Z dist init r=1, world=2 2022-03-17T19:24:56.6403813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:56.6411454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yynj_3k 2022-03-17T19:24:56.6414087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yynj_3k/_remote_module_non_sriptable.py 2022-03-17T19:24:56.6519301Z dist init r=0, world=2 2022-03-17T19:24:56.6523758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:56.6524550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:56.6608531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:58.0207423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:24:58.3326925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:24:58.3327492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:58.3328009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:58.3437545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:58.3438296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:58.3649960Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:58.3650752Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:58.3651898Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:24:58.3652677Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:24:58.4134180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:58.4134851Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:58.4137369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:24:58.4138049Z warnings.warn(msg, FutureWarning) 2022-03-17T19:24:58.4216281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:58.4217087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:24:58.7421798Z ok (2.930s) 2022-03-17T19:24:58.7549077Z test_nested_all_wrapped_model_offload_true_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25388 2022-03-17T19:24:58.7654380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25389 2022-03-17T19:24:59.6202257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdgvuf0d 2022-03-17T19:24:59.6203611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdgvuf0d/_remote_module_non_sriptable.py 2022-03-17T19:24:59.6222769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj27qd45r 2022-03-17T19:24:59.6225828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj27qd45r/_remote_module_non_sriptable.py 2022-03-17T19:24:59.6310160Z dist init r=1, world=2 2022-03-17T19:24:59.6314983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:24:59.6331354Z dist init r=0, world=2 2022-03-17T19:24:59.6335858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:24:59.6337137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:24:59.6418607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:01.0170915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:01.0171442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:01.3264106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:01.3268124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:01.3378377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:01.3378862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:01.3607192Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:01.3608004Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:01.3609156Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:01.3609932Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:01.4107899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:01.4108579Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:01.4110370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:01.4111017Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:01.4188286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:01.4188780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:01.7720730Z ok (3.030s) 2022-03-17T19:25:01.7852571Z test_nested_all_wrapped_model_offload_true_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25475 2022-03-17T19:25:01.7960956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25476 2022-03-17T19:25:02.6195292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74sogf0d 2022-03-17T19:25:02.6196498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74sogf0d/_remote_module_non_sriptable.py 2022-03-17T19:25:02.6316704Z dist init r=0, world=2 2022-03-17T19:25:02.6321023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:02.6460399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4n1u83w 2022-03-17T19:25:02.6463590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4n1u83w/_remote_module_non_sriptable.py 2022-03-17T19:25:02.6565961Z dist init r=1, world=2 2022-03-17T19:25:02.6570476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:02.6571936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:02.6626694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:04.0417863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:04.0418453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:04.3522534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:04.3523334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:04.3632598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:04.3633078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:04.3863013Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:04.3863967Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:04.3865015Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:04.3865774Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:04.4364669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:04.4365340Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:04.4369034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:04.4369692Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:04.4447483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:04.4447988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:04.8026914Z ok (3.030s) 2022-03-17T19:25:04.8154963Z test_nested_all_wrapped_model_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25562 2022-03-17T19:25:04.8259750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25563 2022-03-17T19:25:05.6270186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcq9vqkd9 2022-03-17T19:25:05.6271428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcq9vqkd9/_remote_module_non_sriptable.py 2022-03-17T19:25:05.6376745Z dist init r=1, world=2 2022-03-17T19:25:05.6380882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:05.6613281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_rkap6z 2022-03-17T19:25:05.6615883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_rkap6z/_remote_module_non_sriptable.py 2022-03-17T19:25:05.6719620Z dist init r=0, world=2 2022-03-17T19:25:05.6723966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:05.6725175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:05.6788081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:07.0415984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:07.0416484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:07.3516326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:07.3516795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:07.3625761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:07.3626260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:07.3840929Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:07.3841738Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:07.3842776Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:07.3843539Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:07.4326341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:07.4326989Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:07.4328914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:07.4329522Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:07.4405775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:07.8326649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:07.8326953Z ok (3.030s) 2022-03-17T19:25:07.8455932Z test_nested_all_wrapped_model_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25649 2022-03-17T19:25:07.8559680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25650 2022-03-17T19:25:08.6748936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw3odxi9 2022-03-17T19:25:08.6749894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw3odxi9/_remote_module_non_sriptable.py 2022-03-17T19:25:08.6854861Z dist init r=0, world=2 2022-03-17T19:25:08.6859088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:08.7001595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0jzkerq 2022-03-17T19:25:08.7003974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0jzkerq/_remote_module_non_sriptable.py 2022-03-17T19:25:08.7109197Z dist init r=1, world=2 2022-03-17T19:25:08.7113520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:08.7114463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:08.7164745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:10.0879913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:10.0901631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:10.3977727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:10.3978267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:10.4087205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:10.4087685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:10.4298041Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:10.4298856Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:10.4299904Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:10.4300953Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:10.4776813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:10.4777475Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:10.4779091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:10.4779738Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:10.4856765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:10.4857267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:10.8627110Z ok (3.030s) 2022-03-17T19:25:10.8753313Z test_nested_wrapped_model_offload_false_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25736 2022-03-17T19:25:10.8857751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25737 2022-03-17T19:25:11.6902444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_kvutky 2022-03-17T19:25:11.6903457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_kvutky/_remote_module_non_sriptable.py 2022-03-17T19:25:11.6935434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jif_a5n 2022-03-17T19:25:11.6938058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jif_a5n/_remote_module_non_sriptable.py 2022-03-17T19:25:11.7010014Z dist init r=1, world=2 2022-03-17T19:25:11.7014384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:11.7042222Z dist init r=0, world=2 2022-03-17T19:25:11.7046627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:11.7047404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:11.7117963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:13.0749412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:13.0749950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:13.3920010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:13.3928727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:13.4101293Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:13.4102131Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:13.4107490Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:13.4108275Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:13.4531752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:13.4532413Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:13.4535243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:13.4535921Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:13.4627175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:13.4627682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:13.7921313Z ok (2.929s) 2022-03-17T19:25:13.8046281Z test_nested_wrapped_model_offload_false_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25823 2022-03-17T19:25:13.8150133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25824 2022-03-17T19:25:14.6122570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1abur9ke 2022-03-17T19:25:14.6123750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1abur9ke/_remote_module_non_sriptable.py 2022-03-17T19:25:14.6230540Z dist init r=1, world=2 2022-03-17T19:25:14.6234784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:14.6602629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjuoflzv 2022-03-17T19:25:14.6605258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjuoflzv/_remote_module_non_sriptable.py 2022-03-17T19:25:14.6708761Z dist init r=0, world=2 2022-03-17T19:25:14.6713085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:14.6714295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:14.6742834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:16.0219633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:16.0221406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:16.3327007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:16.3327950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:16.3491900Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:16.3493460Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:16.3495450Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:16.3496957Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:16.3902613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:16.3904061Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:16.3905939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:16.3907232Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:16.3994512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:16.3995449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:16.7216356Z ok (2.929s) 2022-03-17T19:25:16.7342081Z test_nested_wrapped_model_offload_false_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25910 2022-03-17T19:25:16.7446647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25911 2022-03-17T19:25:17.5428678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprihoi7sj 2022-03-17T19:25:17.5429314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprihoi7sj/_remote_module_non_sriptable.py 2022-03-17T19:25:17.5504633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpri2g4s10 2022-03-17T19:25:17.5507701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpri2g4s10/_remote_module_non_sriptable.py 2022-03-17T19:25:17.5535062Z dist init r=1, world=2 2022-03-17T19:25:17.5539390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:17.5610499Z dist init r=0, world=2 2022-03-17T19:25:17.5614624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:17.5615660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:17.5642514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:18.9476066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:18.9476550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:19.2605812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:19.2614272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:19.2787476Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:19.2788577Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:19.2794238Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:19.2795043Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:19.3224417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:19.3225121Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:19.3228543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:19.3229223Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:19.3319573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:19.3320270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:19.6511899Z ok (2.929s) 2022-03-17T19:25:19.6640996Z test_nested_wrapped_model_offload_false_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25997 2022-03-17T19:25:19.6744798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25998 2022-03-17T19:25:20.4691082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgum71a9u 2022-03-17T19:25:20.4691710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgum71a9u/_remote_module_non_sriptable.py 2022-03-17T19:25:20.4798405Z dist init r=1, world=2 2022-03-17T19:25:20.4802599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:20.5209045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6k83xlw5 2022-03-17T19:25:20.5211629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6k83xlw5/_remote_module_non_sriptable.py 2022-03-17T19:25:20.5313936Z dist init r=0, world=2 2022-03-17T19:25:20.5317982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:20.5319031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:20.5412253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:21.8990906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:21.8991403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:22.2090232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:22.2099481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:22.2261704Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:22.2262437Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:22.2268844Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:22.2269612Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:22.2683045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:22.2683636Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:22.2684558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:22.2685271Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:22.2774449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:22.2774949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:22.6813952Z ok (3.030s) 2022-03-17T19:25:22.6945206Z test_nested_wrapped_model_offload_false_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26084 2022-03-17T19:25:22.7052709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26085 2022-03-17T19:25:23.4985000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3brtvey 2022-03-17T19:25:23.4985873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3brtvey/_remote_module_non_sriptable.py 2022-03-17T19:25:23.5031551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylcvqywa 2022-03-17T19:25:23.5034271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylcvqywa/_remote_module_non_sriptable.py 2022-03-17T19:25:23.5091013Z dist init r=1, world=2 2022-03-17T19:25:23.5095516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:23.5138668Z dist init r=0, world=2 2022-03-17T19:25:23.5143482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:23.5144293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:23.5198952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:24.8684720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:24.8685234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:25.1797468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:25.1806392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:25.1980719Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:25.1981556Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:25.1987618Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:25.1988404Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:25.2409456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:25.2410265Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:25.2412226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:25.2412849Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:25.2503523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:25.2504077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:25.6118539Z ok (2.930s) 2022-03-17T19:25:25.6245577Z test_nested_wrapped_model_offload_false_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26171 2022-03-17T19:25:25.6350204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26172 2022-03-17T19:25:26.4267057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjfp2zwx 2022-03-17T19:25:26.4267911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjfp2zwx/_remote_module_non_sriptable.py 2022-03-17T19:25:26.4372758Z dist init r=1, world=2 2022-03-17T19:25:26.4377325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:26.4645870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_qxl7eu 2022-03-17T19:25:26.4648631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_qxl7eu/_remote_module_non_sriptable.py 2022-03-17T19:25:26.4751971Z dist init r=0, world=2 2022-03-17T19:25:26.4756255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:26.4757230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:26.4784298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:27.8380653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:27.8381413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:28.1500984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:28.1509987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:28.1675413Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:28.1676233Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:28.1681870Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:28.1682661Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:28.2096775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:28.2097659Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:28.2098903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:28.2099580Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:28.2192632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:28.2193157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:28.5414805Z ok (2.929s) 2022-03-17T19:25:28.5540228Z test_nested_wrapped_model_offload_true_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26258 2022-03-17T19:25:28.5644825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26259 2022-03-17T19:25:29.3681324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3d2zwv_3 2022-03-17T19:25:29.3682290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3d2zwv_3/_remote_module_non_sriptable.py 2022-03-17T19:25:29.3788270Z dist init r=1, world=2 2022-03-17T19:25:29.3792782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:29.4022645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppcwnt1ju 2022-03-17T19:25:29.4025498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppcwnt1ju/_remote_module_non_sriptable.py 2022-03-17T19:25:29.4127971Z dist init r=0, world=2 2022-03-17T19:25:29.4132237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:29.4133059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:29.4199666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:30.7651065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:30.7651575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:31.0737682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:31.0746808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:31.0863301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:31.0865364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:31.1055610Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:31.1056399Z return iter(self.unbind(0)) 2022-03-17T19:25:31.1057631Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:31.1058425Z return iter(self.unbind(0)) 2022-03-17T19:25:31.1107402Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:31.1108170Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:31.1109526Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:31.1110297Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:31.1600136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:31.1600794Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:31.1601712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:31.1602368Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:31.1691299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:31.1692598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:31.5711996Z ok (3.030s) 2022-03-17T19:25:31.5839049Z test_nested_wrapped_model_offload_true_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26345 2022-03-17T19:25:31.5943119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26346 2022-03-17T19:25:32.4173607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0fquvga 2022-03-17T19:25:32.4174494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0fquvga/_remote_module_non_sriptable.py 2022-03-17T19:25:32.4277867Z dist init r=1, world=2 2022-03-17T19:25:32.4282157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:32.4386300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7y4cougt 2022-03-17T19:25:32.4388964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7y4cougt/_remote_module_non_sriptable.py 2022-03-17T19:25:32.4492801Z dist init r=0, world=2 2022-03-17T19:25:32.4496953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:32.4497895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:32.4588231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:33.8112108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:33.8112876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:34.1241429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:34.1241949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:34.1357726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:34.1358213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:34.1541838Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:34.1542627Z return iter(self.unbind(0)) 2022-03-17T19:25:34.1543760Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:34.1544531Z return iter(self.unbind(0)) 2022-03-17T19:25:34.1591945Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:34.1592744Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:34.1593771Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:34.1594530Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:34.2066549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:34.2067272Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:34.2068480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:34.2069125Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:34.2159004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:34.2159805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:34.6023170Z ok (3.031s) 2022-03-17T19:25:34.6151601Z test_nested_wrapped_model_offload_true_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26432 2022-03-17T19:25:34.6257963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26433 2022-03-17T19:25:35.4180596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rvq3okd 2022-03-17T19:25:35.4181954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rvq3okd/_remote_module_non_sriptable.py 2022-03-17T19:25:35.4286226Z dist init r=1, world=2 2022-03-17T19:25:35.4290761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:35.4830239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwp3ktnrz 2022-03-17T19:25:35.4832604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwp3ktnrz/_remote_module_non_sriptable.py 2022-03-17T19:25:35.4935774Z dist init r=0, world=2 2022-03-17T19:25:35.4940286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:35.4941480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:35.5001416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:36.8608594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:36.8609115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:37.1710935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:37.1711449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:37.1825468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:37.1825942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:37.2018357Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:37.2019063Z return iter(self.unbind(0)) 2022-03-17T19:25:37.2020174Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:37.2021193Z return iter(self.unbind(0)) 2022-03-17T19:25:37.2067913Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:37.2068742Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:37.2069665Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:37.2070714Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:37.2553987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:37.2554580Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:37.2555730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:37.2556293Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:37.2645181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:37.2646021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:37.6325304Z ok (3.030s) 2022-03-17T19:25:37.6450558Z test_nested_wrapped_model_offload_true_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26519 2022-03-17T19:25:37.6554733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26520 2022-03-17T19:25:38.4487477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiukiu8f7 2022-03-17T19:25:38.4488787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiukiu8f7/_remote_module_non_sriptable.py 2022-03-17T19:25:38.4595399Z dist init r=1, world=2 2022-03-17T19:25:38.4599760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:38.4609860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxo6k79i 2022-03-17T19:25:38.4612747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxo6k79i/_remote_module_non_sriptable.py 2022-03-17T19:25:38.4716908Z dist init r=0, world=2 2022-03-17T19:25:38.4721126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:38.4722223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:38.4804551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:39.8340436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:39.8341262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:40.1444362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:40.1454280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:40.1571636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:40.1572132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:40.1761437Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:40.1762993Z return iter(self.unbind(0)) 2022-03-17T19:25:40.1765495Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:40.1767042Z return iter(self.unbind(0)) 2022-03-17T19:25:40.1811859Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:40.1812673Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:40.1813693Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:40.1814440Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:40.2289318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:40.2289997Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:40.2291224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:40.2291874Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:40.2382321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:40.2382834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:40.5619723Z ok (2.929s) 2022-03-17T19:25:40.5747241Z test_nested_wrapped_model_offload_true_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26606 2022-03-17T19:25:40.5851870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26607 2022-03-17T19:25:41.3849902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5brw_32s 2022-03-17T19:25:41.3850780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5brw_32s/_remote_module_non_sriptable.py 2022-03-17T19:25:41.3915029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpysibxkgf 2022-03-17T19:25:41.3917726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpysibxkgf/_remote_module_non_sriptable.py 2022-03-17T19:25:41.3953530Z dist init r=1, world=2 2022-03-17T19:25:41.3957903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:41.4023714Z dist init r=0, world=2 2022-03-17T19:25:41.4027915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:41.4029047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:41.4060789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:42.7714053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:42.7714857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:43.0826659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:43.0827171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:43.0942751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:43.0943250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:43.1139900Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:43.1141047Z return iter(self.unbind(0)) 2022-03-17T19:25:43.1142179Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:43.1142980Z return iter(self.unbind(0)) 2022-03-17T19:25:43.1190792Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:43.1191561Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:43.1192584Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:43.1193341Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:43.1688561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:43.1689252Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:43.1690170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:43.1690832Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:43.1779945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:43.1780614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:43.5918394Z ok (3.030s) 2022-03-17T19:25:43.6045843Z test_nested_wrapped_model_offload_true_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26693 2022-03-17T19:25:43.6149347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26694 2022-03-17T19:25:44.4093210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lx139g9 2022-03-17T19:25:44.4094331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lx139g9/_remote_module_non_sriptable.py 2022-03-17T19:25:44.4199156Z dist init r=1, world=2 2022-03-17T19:25:44.4203430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:44.4564135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0u5jqcp3 2022-03-17T19:25:44.4567682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0u5jqcp3/_remote_module_non_sriptable.py 2022-03-17T19:25:44.4671540Z dist init r=0, world=2 2022-03-17T19:25:44.4676143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:44.4676939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:44.4711263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:45.8572267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:45.8572784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:46.1653912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:46.1654448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:46.1771733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:46.1772230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:46.1959529Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:46.1960341Z return iter(self.unbind(0)) 2022-03-17T19:25:46.1961464Z /opt/conda/lib/python3.7/site-packages/torch/_tensor.py:740: UserWarning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:94.) 2022-03-17T19:25:46.1962233Z return iter(self.unbind(0)) 2022-03-17T19:25:46.2010295Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:46.2011039Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:46.2012043Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:46.2012809Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:46.2490913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:46.2491812Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:46.2493884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:46.2494562Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:46.2586148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:46.2586615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:46.6216457Z ok (3.030s) 2022-03-17T19:25:46.6346393Z test_transformer_parameterized_offload_false_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26780 2022-03-17T19:25:46.6450486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26781 2022-03-17T19:25:47.4457127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7jlv6jg 2022-03-17T19:25:47.4457969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7jlv6jg/_remote_module_non_sriptable.py 2022-03-17T19:25:47.4564147Z dist init r=1, world=2 2022-03-17T19:25:47.4568712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:47.4950964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24w0w6h9 2022-03-17T19:25:47.4953555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24w0w6h9/_remote_module_non_sriptable.py 2022-03-17T19:25:47.5057642Z dist init r=0, world=2 2022-03-17T19:25:47.5062189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:47.5063528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:47.5076456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:48.9075806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:48.9076628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:49.6085626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:49.6086367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:49.6534126Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:49.6534979Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:49.6535989Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:49.6536739Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:49.7046576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:49.7047753Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:49.7049015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:49.7049970Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:49.7609026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:49.7613163Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:50.2527663Z ok (3.631s) 2022-03-17T19:25:50.2655825Z test_transformer_parameterized_offload_false_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26867 2022-03-17T19:25:50.2759855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26868 2022-03-17T19:25:51.0826322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc001la92 2022-03-17T19:25:51.0827311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc001la92/_remote_module_non_sriptable.py 2022-03-17T19:25:51.0932620Z dist init r=1, world=2 2022-03-17T19:25:51.0936920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:51.1179297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhcc0hwo 2022-03-17T19:25:51.1182066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhcc0hwo/_remote_module_non_sriptable.py 2022-03-17T19:25:51.1285767Z dist init r=0, world=2 2022-03-17T19:25:51.1290410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:51.1291145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:51.1344479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:52.4902559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:52.4903118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:53.1822662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:53.1850067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:53.2299292Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:53.2300075Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:53.2305336Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:53.2306118Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:53.2818761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:53.2819430Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:53.2831163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:53.2832147Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:53.3394771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:53.3395255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:53.8836420Z ok (3.631s) 2022-03-17T19:25:53.8965015Z test_transformer_parameterized_offload_false_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26954 2022-03-17T19:25:53.9070142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26955 2022-03-17T19:25:54.7017737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgf1q2eac 2022-03-17T19:25:54.7018777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgf1q2eac/_remote_module_non_sriptable.py 2022-03-17T19:25:54.7123417Z dist init r=0, world=2 2022-03-17T19:25:54.7127955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:54.7171526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_xs_1ou 2022-03-17T19:25:54.7174660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_xs_1ou/_remote_module_non_sriptable.py 2022-03-17T19:25:54.7279985Z dist init r=1, world=2 2022-03-17T19:25:54.7285053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:54.7286331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:54.7333335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:56.1049490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:56.1050316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:25:56.8015030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:56.8015989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:56.8470448Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:56.8471914Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:56.8474041Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:25:56.8475560Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:25:56.8989238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:56.8990505Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:56.9000767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:25:56.9002492Z warnings.warn(msg, FutureWarning) 2022-03-17T19:25:56.9572305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:56.9575651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:25:57.5145454Z ok (3.631s) 2022-03-17T19:25:57.5276766Z test_transformer_parameterized_offload_false_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27041 2022-03-17T19:25:57.5382693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27042 2022-03-17T19:25:58.3602607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcddl7tjo 2022-03-17T19:25:58.3603743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcddl7tjo/_remote_module_non_sriptable.py 2022-03-17T19:25:58.3608466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wf0_90u 2022-03-17T19:25:58.3611324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wf0_90u/_remote_module_non_sriptable.py 2022-03-17T19:25:58.3707785Z dist init r=1, world=2 2022-03-17T19:25:58.3711967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:25:58.3716374Z dist init r=0, world=2 2022-03-17T19:25:58.3720413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:25:58.3721357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:58.3815275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:25:59.7484443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:25:59.7484949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:00.4418627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:00.4419151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:00.4870042Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:00.4870867Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:00.4875688Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:00.4876484Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:00.5382856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:00.5383532Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:00.5395960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:00.5396830Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:00.5960158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:00.5960662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:01.1457250Z ok (3.631s) 2022-03-17T19:26:01.1586424Z test_transformer_parameterized_offload_false_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27128 2022-03-17T19:26:01.1691710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27129 2022-03-17T19:26:01.9686349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphir0z3o9 2022-03-17T19:26:01.9686966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphir0z3o9/_remote_module_non_sriptable.py 2022-03-17T19:26:01.9790454Z dist init r=1, world=2 2022-03-17T19:26:01.9794804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:02.0256495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujwu3b5q 2022-03-17T19:26:02.0259476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujwu3b5q/_remote_module_non_sriptable.py 2022-03-17T19:26:02.0365308Z dist init r=0, world=2 2022-03-17T19:26:02.0370338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:02.0371141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:02.0403915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:03.4292306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:03.4292751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:04.1290185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:04.1290649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:04.1745534Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:04.1746340Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:04.1752073Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:04.1752845Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:04.2262020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:04.2262692Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:04.2279295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:04.2279974Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:04.2865801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:04.2866261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:04.7778401Z ok (3.632s) 2022-03-17T19:26:04.7905495Z test_transformer_parameterized_offload_false_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27215 2022-03-17T19:26:04.8009350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27216 2022-03-17T19:26:05.6100158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiatnt7se 2022-03-17T19:26:05.6101349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiatnt7se/_remote_module_non_sriptable.py 2022-03-17T19:26:05.6204985Z dist init r=0, world=2 2022-03-17T19:26:05.6209218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:05.6458745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5ws5olw 2022-03-17T19:26:05.6461412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5ws5olw/_remote_module_non_sriptable.py 2022-03-17T19:26:05.6565691Z dist init r=1, world=2 2022-03-17T19:26:05.6570121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:05.6570912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:05.6616243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:07.0213536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:07.0214068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:07.7165702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:07.7191277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:07.7635831Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:07.7636615Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:07.7641522Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:07.7642312Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:07.8146227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:07.8146892Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:07.8154029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:07.8154701Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:07.8706767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:07.8710195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:08.4086666Z ok (3.631s) 2022-03-17T19:26:08.4214942Z test_transformer_parameterized_offload_false_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27302 2022-03-17T19:26:08.4319771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27303 2022-03-17T19:26:09.2267839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi27b7cl1 2022-03-17T19:26:09.2269099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi27b7cl1/_remote_module_non_sriptable.py 2022-03-17T19:26:09.2373802Z dist init r=1, world=2 2022-03-17T19:26:09.2378198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:09.2446056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttou0rox 2022-03-17T19:26:09.2448852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttou0rox/_remote_module_non_sriptable.py 2022-03-17T19:26:09.2553389Z dist init r=0, world=2 2022-03-17T19:26:09.2557821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:09.2559020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:09.2582451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:10.6537682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:10.6538168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:11.3869846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:11.3870368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:11.4318964Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:11.4319780Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:11.4320799Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:11.4321578Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:11.4830968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:11.4831646Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:11.4832543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:11.4833206Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:11.5388209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:11.5388688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:12.0396347Z ok (3.631s) 2022-03-17T19:26:12.0525442Z test_transformer_parameterized_offload_false_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27389 2022-03-17T19:26:12.0629937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27390 2022-03-17T19:26:12.8574096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2d1gcefo 2022-03-17T19:26:12.8575053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2d1gcefo/_remote_module_non_sriptable.py 2022-03-17T19:26:12.8680690Z dist init r=1, world=2 2022-03-17T19:26:12.8685054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:12.8716676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63okgfkc 2022-03-17T19:26:12.8719458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63okgfkc/_remote_module_non_sriptable.py 2022-03-17T19:26:12.8824359Z dist init r=0, world=2 2022-03-17T19:26:12.8828874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:12.8829673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:12.8889312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:14.2637627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:14.2638143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:14.9527939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:14.9545783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:14.9991731Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:14.9992535Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:14.9996124Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:14.9996809Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:15.0504493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:15.0505223Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:15.0506021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:15.0506692Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:15.1072798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:15.1073330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:15.6706846Z ok (3.631s) 2022-03-17T19:26:15.6833007Z test_transformer_parameterized_offload_false_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27476 2022-03-17T19:26:15.6937599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27477 2022-03-17T19:26:16.5208600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw345fa5q 2022-03-17T19:26:16.5209291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0go97vhs 2022-03-17T19:26:16.5209831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw345fa5q/_remote_module_non_sriptable.py 2022-03-17T19:26:16.5211152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0go97vhs/_remote_module_non_sriptable.py 2022-03-17T19:26:16.5316031Z dist init r=0, world=2 2022-03-17T19:26:16.5318129Z dist init r=1, world=2 2022-03-17T19:26:16.5320387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:16.5322091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:16.5322909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:16.5424585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:17.9066746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:17.9067261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:18.6083162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:18.6083732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:18.6535259Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:18.6536094Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:18.6540535Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:18.6541332Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:18.7052837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:18.7053519Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:18.7056736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:18.7057413Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:18.7623372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:18.7628368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:19.3015788Z ok (3.631s) 2022-03-17T19:26:19.3143250Z test_transformer_parameterized_offload_false_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27563 2022-03-17T19:26:19.3247814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27564 2022-03-17T19:26:20.1277066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0py_ldb5 2022-03-17T19:26:20.1277678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0py_ldb5/_remote_module_non_sriptable.py 2022-03-17T19:26:20.1381195Z dist init r=0, world=2 2022-03-17T19:26:20.1385719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:20.1714299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrkon22u 2022-03-17T19:26:20.1716844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrkon22u/_remote_module_non_sriptable.py 2022-03-17T19:26:20.1822008Z dist init r=1, world=2 2022-03-17T19:26:20.1827004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:20.1827747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:20.1894377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:21.5471658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:21.5472260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:22.2379743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:22.2380543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:22.2833349Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:22.2834172Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:22.2835202Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:22.2835947Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:22.3347779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:22.3348494Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:22.3354540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:22.3355216Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:22.3923433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:22.3923910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:22.9324529Z ok (3.631s) 2022-03-17T19:26:22.9452634Z test_transformer_parameterized_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27650 2022-03-17T19:26:22.9557881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27651 2022-03-17T19:26:23.7511578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05r2cze4 2022-03-17T19:26:23.7512443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05r2cze4/_remote_module_non_sriptable.py 2022-03-17T19:26:23.7617283Z dist init r=1, world=2 2022-03-17T19:26:23.7621790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:23.7933986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrp285rk 2022-03-17T19:26:23.7936694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrp285rk/_remote_module_non_sriptable.py 2022-03-17T19:26:23.8040154Z dist init r=0, world=2 2022-03-17T19:26:23.8044710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:23.8045684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:23.8129850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:25.1809054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:25.1809622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:25.8791321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:25.8810257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:25.9257132Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:25.9257882Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:25.9263629Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:25.9264348Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:25.9771262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:25.9771966Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:25.9785782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:25.9786440Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:26.0351235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:26.0351757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:26.5637850Z ok (3.631s) 2022-03-17T19:26:26.5767065Z test_transformer_parameterized_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27737 2022-03-17T19:26:26.5871581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27738 2022-03-17T19:26:27.3900075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtm39tvr 2022-03-17T19:26:27.3900888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtm39tvr/_remote_module_non_sriptable.py 2022-03-17T19:26:27.4006085Z dist init r=1, world=2 2022-03-17T19:26:27.4010447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:27.4168216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpted52y3v 2022-03-17T19:26:27.4171820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpted52y3v/_remote_module_non_sriptable.py 2022-03-17T19:26:27.4274207Z dist init r=0, world=2 2022-03-17T19:26:27.4278569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:27.4279332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:27.4316001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:28.7961381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:28.7961900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:29.4871104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:29.4895771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:29.5339622Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:29.5340676Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:29.5345630Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:29.5346315Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:29.5849495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:29.5850310Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:29.5851251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:29.5851795Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:29.6416140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:29.6416639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:30.1948036Z ok (3.631s) 2022-03-17T19:26:30.2073810Z test_transformer_parameterized_offload_true_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27824 2022-03-17T19:26:30.2177915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27825 2022-03-17T19:26:31.0212910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht8cuedp 2022-03-17T19:26:31.0213729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht8cuedp/_remote_module_non_sriptable.py 2022-03-17T19:26:31.0247572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8w7uz0o 2022-03-17T19:26:31.0250188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8w7uz0o/_remote_module_non_sriptable.py 2022-03-17T19:26:31.0318109Z dist init r=1, world=2 2022-03-17T19:26:31.0322304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:31.0356133Z dist init r=0, world=2 2022-03-17T19:26:31.0360238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:31.0361290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:31.0425916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:32.4238417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:32.4238919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:33.1175347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:33.1175841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:33.1490443Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:33.1491737Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:33.1783643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:33.1784088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:33.2276231Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:33.2276985Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:33.2277992Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:33.2278774Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:33.2801679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:33.2802314Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:33.2803217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:33.2804195Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:33.3365808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:33.3366293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:33.8255320Z ok (3.631s) 2022-03-17T19:26:33.8381701Z test_transformer_parameterized_offload_true_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27911 2022-03-17T19:26:33.8485238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27912 2022-03-17T19:26:34.6677471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdw58apy 2022-03-17T19:26:34.6678319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdw58apy/_remote_module_non_sriptable.py 2022-03-17T19:26:34.6692446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp297fpyyx 2022-03-17T19:26:34.6695339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp297fpyyx/_remote_module_non_sriptable.py 2022-03-17T19:26:34.6783688Z dist init r=0, world=2 2022-03-17T19:26:34.6788331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:34.6801122Z dist init r=1, world=2 2022-03-17T19:26:34.6805585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:34.6806475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:34.6891848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:36.0602769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:36.0603285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:36.7676476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:36.7677026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:36.7997510Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:36.7999140Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:36.8291709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:36.8292318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:36.8794821Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:36.8795651Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:36.8796680Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:36.8797747Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:36.9325788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:36.9326479Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:36.9334226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:36.9334970Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:36.9914718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:36.9915228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:37.5565448Z ok (3.731s) 2022-03-17T19:26:37.5693274Z test_transformer_parameterized_offload_true_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27998 2022-03-17T19:26:37.5797607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27999 2022-03-17T19:26:38.3737069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpbhj7l3 2022-03-17T19:26:38.3737665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpbhj7l3/_remote_module_non_sriptable.py 2022-03-17T19:26:38.3842825Z dist init r=1, world=2 2022-03-17T19:26:38.3847133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:38.4257984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5314kw0 2022-03-17T19:26:38.4260804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5314kw0/_remote_module_non_sriptable.py 2022-03-17T19:26:38.4363199Z dist init r=0, world=2 2022-03-17T19:26:38.4367538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:38.4368619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:38.4456640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:39.8078871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:39.8079422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:40.5059218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:40.5059734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:40.5371682Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:40.5373638Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:40.5659412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:40.5662252Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:40.6147871Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:40.6148702Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:40.6150306Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:40.6151286Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:40.6677658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:40.6678321Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:40.6692446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:40.6693231Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:40.7249406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:40.7252019Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:41.2877095Z ok (3.731s) 2022-03-17T19:26:41.3007349Z test_transformer_parameterized_offload_true_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28085 2022-03-17T19:26:41.3112442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28086 2022-03-17T19:26:42.1236375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsteod761 2022-03-17T19:26:42.1237133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsteod761/_remote_module_non_sriptable.py 2022-03-17T19:26:42.1341724Z dist init r=1, world=2 2022-03-17T19:26:42.1346715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:42.1477965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa86wdj8z 2022-03-17T19:26:42.1480672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa86wdj8z/_remote_module_non_sriptable.py 2022-03-17T19:26:42.1586254Z dist init r=0, world=2 2022-03-17T19:26:42.1591157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:42.1592117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:42.1652048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:43.5227719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:43.5228198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:44.2234957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:44.2235812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:44.2548437Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:44.2556952Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:44.2850376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:44.2850886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:44.3349238Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:44.3349997Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:44.3351032Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:44.3351795Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:44.3882225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:44.3882906Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:44.3885979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:44.3886621Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:44.4459020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:44.4459513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:45.0192314Z ok (3.731s) 2022-03-17T19:26:45.0323325Z test_transformer_parameterized_offload_true_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28172 2022-03-17T19:26:45.0429489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28173 2022-03-17T19:26:45.8370102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezyasc8s 2022-03-17T19:26:45.8371079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezyasc8s/_remote_module_non_sriptable.py 2022-03-17T19:26:45.8477569Z dist init r=1, world=2 2022-03-17T19:26:45.8482112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:45.8938225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mmqftrp 2022-03-17T19:26:45.8941080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mmqftrp/_remote_module_non_sriptable.py 2022-03-17T19:26:45.9045332Z dist init r=0, world=2 2022-03-17T19:26:45.9050152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:45.9050938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:45.9091754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:47.2935846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:47.2936395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:47.9907633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:47.9909967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:48.0226081Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:48.0227418Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:48.0517883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:48.0518362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:48.1010321Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:48.1011073Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:48.1012089Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:48.1012844Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:48.1545729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:48.1546343Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:48.1547510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:48.1548156Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:48.2110734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:48.2127425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:48.7508063Z ok (3.731s) 2022-03-17T19:26:48.7637625Z test_transformer_parameterized_offload_true_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28259 2022-03-17T19:26:48.7742692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28260 2022-03-17T19:26:49.5696931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fda2gjf 2022-03-17T19:26:49.5698123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fda2gjf/_remote_module_non_sriptable.py 2022-03-17T19:26:49.5743463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd8fzy0f 2022-03-17T19:26:49.5746491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd8fzy0f/_remote_module_non_sriptable.py 2022-03-17T19:26:49.5802925Z dist init r=1, world=2 2022-03-17T19:26:49.5807426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:49.5854182Z dist init r=0, world=2 2022-03-17T19:26:49.5859161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:49.5860884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:49.5911405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:50.9482804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:50.9483751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:51.6413599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:51.6440283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:51.6727330Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:51.6758334Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:51.7040641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:51.7047747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:51.7538317Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:51.7539934Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:51.7542401Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:51.7543936Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:51.8071886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:51.8073277Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:51.8078073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:51.8079404Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:51.8639218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:51.8647460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:52.3820237Z ok (3.631s) 2022-03-17T19:26:52.3949350Z test_transformer_parameterized_offload_true_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28346 2022-03-17T19:26:52.4053499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28347 2022-03-17T19:26:53.2042454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iufpdav 2022-03-17T19:26:53.2044094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iufpdav/_remote_module_non_sriptable.py 2022-03-17T19:26:53.2152340Z dist init r=1, world=2 2022-03-17T19:26:53.2157170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:53.2369229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwz5yaka 2022-03-17T19:26:53.2371785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwz5yaka/_remote_module_non_sriptable.py 2022-03-17T19:26:53.2474355Z dist init r=0, world=2 2022-03-17T19:26:53.2478527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:53.2479753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:53.2564394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:54.6029313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:54.6029856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:55.2950319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:55.2972754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:55.3265174Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:55.3298284Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:55.3589927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:55.3597173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:55.4094634Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:55.4096160Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:55.4098234Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:55.4100154Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:55.4631505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:55.4632889Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:55.4650994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:55.4652356Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:55.5231562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:55.5244710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:56.0131405Z ok (3.631s) 2022-03-17T19:26:56.0260163Z test_transformer_parameterized_offload_true_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28433 2022-03-17T19:26:56.0365006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28434 2022-03-17T19:26:56.8319386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7ni6zdw 2022-03-17T19:26:56.8320502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7ni6zdw/_remote_module_non_sriptable.py 2022-03-17T19:26:56.8427906Z dist init r=1, world=2 2022-03-17T19:26:56.8432557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:26:56.8689222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4b0dz8d 2022-03-17T19:26:56.8691652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4b0dz8d/_remote_module_non_sriptable.py 2022-03-17T19:26:56.8794056Z dist init r=0, world=2 2022-03-17T19:26:56.8798916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:26:56.8799614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:56.8839686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:26:58.2603089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:26:58.2604060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:26:58.9518853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:58.9539040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:58.9830016Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:58.9861346Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:26:59.0146810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:59.0153175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:59.0642124Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:59.0643896Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:59.0646011Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:26:59.0647523Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:26:59.1172906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:59.1174163Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:59.1184442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:26:59.1185816Z warnings.warn(msg, FutureWarning) 2022-03-17T19:26:59.1746073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:59.1753591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:26:59.6440765Z ok (3.631s) 2022-03-17T19:26:59.6568575Z test_transformer_parameterized_offload_true_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28520 2022-03-17T19:26:59.6672281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28521 2022-03-17T19:27:00.4600410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkc4c2q1 2022-03-17T19:27:00.4601309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkc4c2q1/_remote_module_non_sriptable.py 2022-03-17T19:27:00.4671312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkdmr9wy 2022-03-17T19:27:00.4673997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkdmr9wy/_remote_module_non_sriptable.py 2022-03-17T19:27:00.4706133Z dist init r=0, world=2 2022-03-17T19:27:00.4710523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:00.4778793Z dist init r=1, world=2 2022-03-17T19:27:00.4783537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:00.4784538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:00.4813560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:01.8539434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:01.8539961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:02.5486640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:02.5506794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:02.5799049Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:27:02.5828619Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:27:02.6111517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:02.6117591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:02.6609325Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:27:02.6610143Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:27:02.6611875Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:27:02.6612635Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:27:02.7139807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:27:02.7140804Z warnings.warn(msg, FutureWarning) 2022-03-17T19:27:02.7143176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:27:02.7143842Z warnings.warn(msg, FutureWarning) 2022-03-17T19:27:02.7701359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:02.7706729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:03.2747668Z ok (3.631s) 2022-03-17T19:27:03.2876758Z test_transformer_parameterized_offload_true_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28607 2022-03-17T19:27:03.2980729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28608 2022-03-17T19:27:04.0993470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwiz6kdy 2022-03-17T19:27:04.0994652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwiz6kdy/_remote_module_non_sriptable.py 2022-03-17T19:27:04.1100725Z dist init r=1, world=2 2022-03-17T19:27:04.1105230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:04.1336551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxz89b_2 2022-03-17T19:27:04.1339356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxz89b_2/_remote_module_non_sriptable.py 2022-03-17T19:27:04.1444074Z dist init r=0, world=2 2022-03-17T19:27:04.1448461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:04.1449236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:04.1512106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:05.5207747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:05.5208629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:06.2109776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:06.2110301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:06.2416968Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:27:06.2432172Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:27:06.2712250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:06.2725313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:06.3222098Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:27:06.3222951Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:27:06.3223947Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:27:06.3224701Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:27:06.3747377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:27:06.3748036Z warnings.warn(msg, FutureWarning) 2022-03-17T19:27:06.3756339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:27:06.3757002Z warnings.warn(msg, FutureWarning) 2022-03-17T19:27:06.4321477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:06.4334517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:07.0060407Z ok (3.731s) 2022-03-17T19:27:07.0189393Z test_transformer_parameterized_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28694 2022-03-17T19:27:07.0293205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28695 2022-03-17T19:27:07.8237164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmces9e0 2022-03-17T19:27:07.8238104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmces9e0/_remote_module_non_sriptable.py 2022-03-17T19:27:07.8344332Z dist init r=0, world=2 2022-03-17T19:27:07.8348708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:07.8786304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglbiscmm 2022-03-17T19:27:07.8788987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglbiscmm/_remote_module_non_sriptable.py 2022-03-17T19:27:07.8891842Z dist init r=1, world=2 2022-03-17T19:27:07.8896212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:07.8897362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:07.8958026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:09.2602450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:09.2602965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:09.9506424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:09.9509898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:09.9828702Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:27:09.9830279Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:27:10.0119200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:10.0119666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:10.0612230Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:27:10.0613005Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:27:10.0614010Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:27:10.0614760Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:27:10.1139572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:27:10.1140830Z warnings.warn(msg, FutureWarning) 2022-03-17T19:27:10.1141748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:27:10.1142396Z warnings.warn(msg, FutureWarning) 2022-03-17T19:27:10.1705664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:10.1706245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:10.7370364Z ok (3.731s) 2022-03-17T19:27:10.7500882Z test_transformer_parameterized_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28781 2022-03-17T19:27:10.7605394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28782 2022-03-17T19:27:11.5732497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigof8sa_ 2022-03-17T19:27:11.5733662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigof8sa_/_remote_module_non_sriptable.py 2022-03-17T19:27:11.5838540Z dist init r=1, world=2 2022-03-17T19:27:11.5842932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:11.6024349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_na59mb 2022-03-17T19:27:11.6026762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_na59mb/_remote_module_non_sriptable.py 2022-03-17T19:27:11.6128485Z dist init r=0, world=2 2022-03-17T19:27:11.6132771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:11.6133781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:11.6148077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:12.9668484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:12.9669025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:13.6582691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:13.6597304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:13.6895060Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:27:13.6919268Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:27:13.7205900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:13.7210897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:13.7706015Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:27:13.7707125Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:27:13.7708466Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:27:13.7709218Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:27:13.8233570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:27:13.8234294Z warnings.warn(msg, FutureWarning) 2022-03-17T19:27:13.8239337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844. 2022-03-17T19:27:13.8239961Z warnings.warn(msg, FutureWarning) 2022-03-17T19:27:13.8807460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:13.8812338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:14.3699974Z ok (3.633s) 2022-03-17T19:27:14.3700530Z 2022-03-17T19:27:14.3700947Z ---------------------------------------------------------------------- 2022-03-17T19:27:14.3701290Z Ran 127 tests in 475.008s 2022-03-17T19:27:14.3701454Z 2022-03-17T19:27:14.3701548Z OK 2022-03-17T19:27:14.3701664Z 2022-03-17T19:27:14.3701796Z Generating XML reports... 2022-03-17T19:27:14.3752569Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20220317191919.xml 2022-03-17T19:27:14.3755937Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20220317191919.xml 2022-03-17T19:27:14.3759459Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20220317191919.xml 2022-03-17T19:27:14.3900603Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20220317191919.xml 2022-03-17T19:27:14.6244284Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-03-17 19:27:14.623915] 2022-03-17T19:27:14.6244978Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:27:14.624021] 2022-03-17T19:27:15.4478890Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights 2022-03-17T19:27:15.4496400Z 2022-03-17T19:27:15.4496798Z Running tests... 2022-03-17T19:27:15.4497278Z ---------------------------------------------------------------------- 2022-03-17T19:27:16.9971127Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:27:17.0387829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28905 2022-03-17T19:27:17.0509517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28906 2022-03-17T19:27:17.8709835Z dist init r=0, world=2 2022-03-17T19:27:17.8713224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:17.8811966Z dist init r=1, world=2 2022-03-17T19:27:17.8816541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:17.8818035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:17.8917622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:19.2421206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:19.2421652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:20.2766586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:20.2767078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:20.7598390Z ok (5.310s) 2022-03-17T19:27:20.7749574Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28992 2022-03-17T19:27:20.7869256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28993 2022-03-17T19:27:21.5920994Z dist init r=1, world=2 2022-03-17T19:27:21.5924345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:21.6240956Z dist init r=0, world=2 2022-03-17T19:27:21.6245976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:21.6246805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:21.6331133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:23.0132096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:23.0132723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:24.0442807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:24.0443336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:24.5952917Z ok (3.835s) 2022-03-17T19:27:24.6106721Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29079 2022-03-17T19:27:24.6226536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29080 2022-03-17T19:27:25.4257817Z dist init r=1, world=2 2022-03-17T19:27:25.4261204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:25.4632081Z dist init r=0, world=2 2022-03-17T19:27:25.4636540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:25.4637731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:25.4668475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:26.8186987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:26.8187685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:27.7832396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:27.7835068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:28.2304981Z ok (3.635s) 2022-03-17T19:27:28.2458288Z test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29166 2022-03-17T19:27:28.2573344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29167 2022-03-17T19:27:29.0669685Z dist init r=1, world=2 2022-03-17T19:27:29.0672715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:29.0775392Z dist init r=0, world=2 2022-03-17T19:27:29.0780574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:29.0781609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:29.0877546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:30.4485264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:30.4485813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:31.4213871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:31.4214410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:31.9653900Z ok (3.735s) 2022-03-17T19:27:31.9810032Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29253 2022-03-17T19:27:31.9930372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29254 2022-03-17T19:27:32.8543846Z dist init r=1, world=2 2022-03-17T19:27:32.8547782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:32.8556597Z dist init r=0, world=2 2022-03-17T19:27:32.8561397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:32.8562367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:32.8650606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:34.2416160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:34.2416679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:35.3383205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:35.3383743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:35.9012658Z ok (3.936s) 2022-03-17T19:27:35.9163420Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29340 2022-03-17T19:27:35.9281980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29341 2022-03-17T19:27:36.7844411Z dist init r=0, world=2 2022-03-17T19:27:36.7844745Z dist init r=1, world=2 2022-03-17T19:27:36.7848133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:36.7848577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:36.7849449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:36.7850335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:38.1439804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:38.1440305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:39.1975447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:39.1975954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:39.7364023Z ok (3.835s) 2022-03-17T19:27:39.7512991Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29457 2022-03-17T19:27:39.7630310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29458 2022-03-17T19:27:40.5721883Z dist init r=1, world=2 2022-03-17T19:27:40.5724616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:40.5837302Z dist init r=0, world=2 2022-03-17T19:27:40.5842029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:40.5843022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:40.5929179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:41.9546294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:41.9546810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:42.9185766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:42.9188593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:43.3710401Z ok (3.634s) 2022-03-17T19:27:43.3856331Z test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29544 2022-03-17T19:27:43.3972681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29545 2022-03-17T19:27:44.2145971Z dist init r=1, world=2 2022-03-17T19:27:44.2149424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:44.2418431Z dist init r=0, world=2 2022-03-17T19:27:44.2423170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:44.2424352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:44.2455098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:45.6147827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:45.6148521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:46.5838953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:46.5839489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:27:47.1053969Z ok (3.734s) 2022-03-17T19:27:47.1054258Z 2022-03-17T19:27:47.1054789Z ---------------------------------------------------------------------- 2022-03-17T19:27:47.1055160Z Ran 8 tests in 31.656s 2022-03-17T19:27:47.1055342Z 2022-03-17T19:27:47.1055420Z OK 2022-03-17T19:27:47.1055557Z 2022-03-17T19:27:47.1058806Z Generating XML reports... 2022-03-17T19:27:47.1111539Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220317192715.xml 2022-03-17T19:27:47.3506843Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-03-17 19:27:47.350155] 2022-03-17T19:27:47.3507721Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:27:47.350261] 2022-03-17T19:27:48.1802747Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc 2022-03-17T19:27:48.1820690Z 2022-03-17T19:27:48.1820917Z Running tests... 2022-03-17T19:27:48.1821374Z ---------------------------------------------------------------------- 2022-03-17T19:27:48.1838561Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc) 2022-03-17T19:27:49.7177393Z Tests gradient accumulation. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:27:49.7595985Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29698 2022-03-17T19:27:49.7716497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29699 2022-03-17T19:27:50.5687164Z dist init r=0, world=2 2022-03-17T19:27:50.5691626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:50.5754440Z dist init r=1, world=2 2022-03-17T19:27:50.5760412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:50.5761243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:50.5794488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:51.9406837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:51.9407364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:52.6793792Z ok (4.497s) 2022-03-17T19:27:52.6809527Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc) 2022-03-17T19:27:52.6946508Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29785 2022-03-17T19:27:52.7066019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29786 2022-03-17T19:27:53.5301587Z dist init r=1, world=2 2022-03-17T19:27:53.5306042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:53.5428206Z dist init r=0, world=2 2022-03-17T19:27:53.5433856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:53.5434950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:53.5510527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:54.9096335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:54.9096832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:55.7133099Z ok (3.034s) 2022-03-17T19:27:55.7148989Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc) 2022-03-17T19:27:55.7288999Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29872 2022-03-17T19:27:55.7405465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29873 2022-03-17T19:27:56.5463064Z dist init r=1, world=2 2022-03-17T19:27:56.5468154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:56.5503406Z dist init r=0, world=2 2022-03-17T19:27:56.5509275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:56.5510116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:56.5570970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:57.8990941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:27:57.8991736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:27:58.6471687Z ok (2.934s) 2022-03-17T19:27:58.6486975Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc) 2022-03-17T19:27:58.6627180Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29959 2022-03-17T19:27:58.6745625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29960 2022-03-17T19:27:59.4809773Z dist init r=1, world=2 2022-03-17T19:27:59.4814505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:27:59.5231773Z dist init r=0, world=2 2022-03-17T19:27:59.5237661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:27:59.5238482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:27:59.5322324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:00.8944035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:00.8944594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:01.1803694Z ok (2.533s) 2022-03-17T19:28:01.1818815Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc) 2022-03-17T19:28:01.1958841Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30042 2022-03-17T19:28:01.2074334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30043 2022-03-17T19:28:02.0330045Z dist init r=1, world=2 2022-03-17T19:28:02.0334457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:02.0684252Z dist init r=0, world=2 2022-03-17T19:28:02.0689997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:02.0690997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:02.0741553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:03.4466271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:03.4466829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:03.7145025Z ok (2.534s) 2022-03-17T19:28:03.7160086Z test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc) 2022-03-17T19:28:03.7298372Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30125 2022-03-17T19:28:03.7414440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30126 2022-03-17T19:28:04.5475004Z dist init r=1, world=2 2022-03-17T19:28:04.5479976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:04.5580112Z dist init r=0, world=2 2022-03-17T19:28:04.5586605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:04.5587408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:04.5684595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:05.9215216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:05.9215729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:06.2471006Z ok (2.533s) 2022-03-17T19:28:06.2486297Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc) 2022-03-17T19:28:06.2623504Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30208 2022-03-17T19:28:06.2738839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30209 2022-03-17T19:28:07.1373339Z dist init r=1, world=2 2022-03-17T19:28:07.1377992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:07.1401340Z dist init r=0, world=2 2022-03-17T19:28:07.1407840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:07.1408649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:07.1480731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:08.4961288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:08.4961814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:09.2808165Z ok (3.034s) 2022-03-17T19:28:09.2825127Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc) 2022-03-17T19:28:09.2965487Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30295 2022-03-17T19:28:09.3082997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30296 2022-03-17T19:28:10.1466817Z dist init r=1, world=2 2022-03-17T19:28:10.1470733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:10.1602249Z dist init r=0, world=2 2022-03-17T19:28:10.1608184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:10.1609343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:10.1675143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:11.5341921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:11.5342445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:12.2148166Z ok (2.934s) 2022-03-17T19:28:12.2164281Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc) 2022-03-17T19:28:12.2303374Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30382 2022-03-17T19:28:12.2418773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30383 2022-03-17T19:28:13.0721495Z dist init r=1, world=2 2022-03-17T19:28:13.0725948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:13.0745843Z dist init r=0, world=2 2022-03-17T19:28:13.0751953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:13.0753007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:13.0829055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:14.4413993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:14.4414517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:15.1485900Z ok (2.934s) 2022-03-17T19:28:15.1501702Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc) 2022-03-17T19:28:15.1639655Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30469 2022-03-17T19:28:15.1756015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30470 2022-03-17T19:28:15.9916935Z dist init r=1, world=2 2022-03-17T19:28:15.9921225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:16.0319470Z dist init r=0, world=2 2022-03-17T19:28:16.0325073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:16.0326091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:16.0327259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:17.3874507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:17.3875021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:17.6815869Z ok (2.533s) 2022-03-17T19:28:17.6831295Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc) 2022-03-17T19:28:17.6969233Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30552 2022-03-17T19:28:17.7084185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30553 2022-03-17T19:28:18.5656306Z dist init r=1, world=2 2022-03-17T19:28:18.5661776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:18.5754390Z dist init r=0, world=2 2022-03-17T19:28:18.5760402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:18.5761684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:18.5764062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:19.9506281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:19.9507161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:20.2141335Z ok (2.532s) 2022-03-17T19:28:20.2157018Z test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc) 2022-03-17T19:28:20.2294794Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30635 2022-03-17T19:28:20.2414615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30636 2022-03-17T19:28:21.0894595Z dist init r=1, world=2 2022-03-17T19:28:21.0898886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:21.0988468Z dist init r=0, world=2 2022-03-17T19:28:21.0994297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:21.0995465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:21.1001221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:22.4589271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:22.4589818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:22.7473151Z ok (2.533s) 2022-03-17T19:28:22.7473377Z 2022-03-17T19:28:22.7476966Z ---------------------------------------------------------------------- 2022-03-17T19:28:22.7477395Z Ran 12 tests in 34.565s 2022-03-17T19:28:22.7477564Z 2022-03-17T19:28:22.7477647Z OK 2022-03-17T19:28:22.7477783Z 2022-03-17T19:28:22.7477927Z Generating XML reports... 2022-03-17T19:28:22.7538302Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220317192748.xml 2022-03-17T19:28:22.9907293Z Running distributed/fsdp/test_fsdp_input ... [2022-03-17 19:28:22.990212] 2022-03-17T19:28:22.9908018Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:28:22.990318] 2022-03-17T19:28:23.8141389Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input 2022-03-17T19:28:23.8157169Z 2022-03-17T19:28:23.8157431Z Running tests... 2022-03-17T19:28:23.8157876Z ---------------------------------------------------------------------- 2022-03-17T19:28:23.8176696Z test_input_type_dict (__main__.TestInput) 2022-03-17T19:28:25.3601306Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:28:25.4025045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30755 2022-03-17T19:28:26.2136456Z dist init r=0, world=1 2022-03-17T19:28:26.2139794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:26.2141406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:28:27.5004556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:28.0089955Z ok (4.193s) 2022-03-17T19:28:28.0108293Z test_input_type_list (__main__.TestInput) 2022-03-17T19:28:28.0246214Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30799 2022-03-17T19:28:28.8303051Z dist init r=0, world=1 2022-03-17T19:28:28.8306804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:28.8307778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:28:30.1223040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:30.6302742Z ok (2.621s) 2022-03-17T19:28:30.6302964Z 2022-03-17T19:28:30.6303343Z ---------------------------------------------------------------------- 2022-03-17T19:28:30.6303662Z Ran 2 tests in 6.814s 2022-03-17T19:28:30.6303826Z 2022-03-17T19:28:30.6304933Z OK 2022-03-17T19:28:30.6305076Z 2022-03-17T19:28:30.6305188Z Generating XML reports... 2022-03-17T19:28:30.6350225Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220317192823.xml 2022-03-17T19:28:30.8730533Z Running distributed/fsdp/test_fsdp_memory ... [2022-03-17 19:28:30.872540] 2022-03-17T19:28:30.8731532Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:28:30.872648] 2022-03-17T19:28:31.7004180Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory 2022-03-17T19:28:31.7021283Z 2022-03-17T19:28:31.7021684Z Running tests... 2022-03-17T19:28:31.7022151Z ---------------------------------------------------------------------- 2022-03-17T19:28:33.2417674Z test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:28:33.2835333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30880 2022-03-17T19:28:33.2962861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30881 2022-03-17T19:28:34.1347147Z dist init r=1, world=2 2022-03-17T19:28:34.1351216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:34.1485172Z dist init r=0, world=2 2022-03-17T19:28:34.1491497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:34.1492454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:34.1555639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:35.5344099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:35.5344615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:39.3080600Z ok (7.606s) 2022-03-17T19:28:39.3254604Z test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30997 2022-03-17T19:28:39.3369403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30998 2022-03-17T19:28:40.1404472Z dist init r=1, world=2 2022-03-17T19:28:40.1409001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:40.1579298Z dist init r=0, world=2 2022-03-17T19:28:40.1585431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:40.1586483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:40.1612989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:41.5271658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:41.5272225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:44.7474502Z ok (5.439s) 2022-03-17T19:28:44.7474857Z 2022-03-17T19:28:44.7475212Z ---------------------------------------------------------------------- 2022-03-17T19:28:44.7475552Z Ran 2 tests in 13.045s 2022-03-17T19:28:44.7475689Z 2022-03-17T19:28:44.7475763Z OK 2022-03-17T19:28:44.7475896Z 2022-03-17T19:28:44.7476028Z Generating XML reports... 2022-03-17T19:28:44.7519978Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220317192831.xml 2022-03-17T19:28:44.9877702Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-03-17 19:28:44.987288] 2022-03-17T19:28:44.9878490Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:28:44.987392] 2022-03-17T19:28:45.8023230Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward 2022-03-17T19:28:45.8039229Z 2022-03-17T19:28:45.8039640Z Running tests... 2022-03-17T19:28:45.8040358Z ---------------------------------------------------------------------- 2022-03-17T19:28:47.3301321Z test_multi_forward (__main__.TestMultiForward) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:28:47.3718430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31151 2022-03-17T19:28:47.3844018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31152 2022-03-17T19:28:48.2121533Z dist init r=0, world=2 2022-03-17T19:28:48.2125148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:48.2317284Z dist init r=1, world=2 2022-03-17T19:28:48.2321892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:48.2322764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:48.2328833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:49.6029848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:49.6030352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:49.9140391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:28:49.9141170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:28:50.1910773Z ok (4.387s) 2022-03-17T19:28:50.1911006Z 2022-03-17T19:28:50.1911426Z ---------------------------------------------------------------------- 2022-03-17T19:28:50.1911747Z Ran 1 test in 4.387s 2022-03-17T19:28:50.1911881Z 2022-03-17T19:28:50.1912000Z OK 2022-03-17T19:28:50.1912092Z 2022-03-17T19:28:50.1912226Z Generating XML reports... 2022-03-17T19:28:50.1954662Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220317192845.xml 2022-03-17T19:28:50.4313316Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-03-17 19:28:50.430857] 2022-03-17T19:28:50.4314058Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:28:50.430961] 2022-03-17T19:28:51.2253188Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping 2022-03-17T19:28:51.2269646Z 2022-03-17T19:28:51.2269896Z Running tests... 2022-03-17T19:28:51.2270327Z ---------------------------------------------------------------------- 2022-03-17T19:28:51.2285218Z test_multiple_wrapping (__main__.TestMultipleWrapping) 2022-03-17T19:28:52.7648088Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:28:52.8070089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31275 2022-03-17T19:28:52.8194693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31276 2022-03-17T19:28:53.6066779Z dist init r=1, world=2 2022-03-17T19:28:53.6070102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:28:53.6095240Z dist init r=0, world=2 2022-03-17T19:28:53.6099734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:53.6101224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:53.6173197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:28:54.9688129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:28:54.9688687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:28:55.6264577Z ok (4.399s) 2022-03-17T19:28:55.6264841Z 2022-03-17T19:28:55.6265749Z ---------------------------------------------------------------------- 2022-03-17T19:28:55.6266092Z Ran 1 test in 4.399s 2022-03-17T19:28:55.6266241Z 2022-03-17T19:28:55.6266359Z OK 2022-03-17T19:28:55.6267359Z 2022-03-17T19:28:55.6267817Z Generating XML reports... 2022-03-17T19:28:55.6310337Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220317192851.xml 2022-03-17T19:28:55.8635110Z Running distributed/fsdp/test_fsdp_overlap ... [2022-03-17 19:28:55.863007] 2022-03-17T19:28:55.8635870Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:28:55.863110] 2022-03-17T19:28:56.6848703Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap 2022-03-17T19:28:56.6866361Z 2022-03-17T19:28:56.6866880Z Running tests... 2022-03-17T19:28:56.6867350Z ---------------------------------------------------------------------- 2022-03-17T19:28:58.2253751Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:28:58.2672357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31399 2022-03-17T19:28:59.0778041Z dist init r=0, world=1 2022-03-17T19:28:59.0780916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:28:59.0781747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:29:00.3737019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:06.1714034Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:29:06.1715350Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:29:06.1716631Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:29:06.1717886Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:29:10.1877294Z 2022-03-17T19:29:10.1877550Z rank0: 2022-03-17T19:29:10.1878153Z e1: {'cpu_iter': 0.001771913600000019, 'cpu_wait': 3.539979999995779e-05, 'gpu_compute': 0.0738943999633193, 'gpu_total': 0.9873088002204895} 2022-03-17T19:29:10.1878972Z e2: {'cpu_iter': 0.0035012666000000083, 'cpu_wait': 3.420020000000079e-05, 'gpu_compute': 0.255231998488307, 'gpu_total': 1.741763210296631} 2022-03-17T19:29:10.1879529Z e3: {'cpu_iter': 0.0018800796000003396, 'cpu_wait': 0.23438227629999986, 'gpu_compute': 235.85241928100587, 'gpu_total': 236.21362762451173} 2022-03-17T19:29:10.1880085Z e4: {'cpu_iter': 0.0035851336000000344, 'cpu_wait': 0.23270960129999985, 'gpu_compute': 235.835196685791, 'gpu_total': 236.3093246459961} 2022-03-17T19:29:10.3869119Z ok (13.700s) 2022-03-17T19:29:10.3877969Z test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip (0.001s) 2022-03-17T19:29:10.3878229Z 2022-03-17T19:29:10.3878885Z ---------------------------------------------------------------------- 2022-03-17T19:29:10.3879236Z Ran 2 tests in 13.701s 2022-03-17T19:29:10.3879391Z 2022-03-17T19:29:10.3879502Z OK (skipped=1) 2022-03-17T19:29:10.3879658Z 2022-03-17T19:29:10.3879783Z Generating XML reports... 2022-03-17T19:29:10.3921364Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220317192856.xml 2022-03-17T19:29:10.3925727Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220317192856.xml 2022-03-17T19:29:10.6249305Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-03-17 19:29:10.624434] 2022-03-17T19:29:10.6250001Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:29:10.624537] 2022-03-17T19:29:11.4452943Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16 2022-03-17T19:29:11.4468809Z 2022-03-17T19:29:11.4469131Z Running tests... 2022-03-17T19:29:11.4469518Z ---------------------------------------------------------------------- 2022-03-17T19:29:12.9889226Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:29:13.0044918Z skip (1.557s) 2022-03-17T19:29:13.0322655Z test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31480 2022-03-17T19:29:13.0445017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31481 2022-03-17T19:29:13.8450329Z dist init r=0, world=2 2022-03-17T19:29:13.8453989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:13.8840306Z dist init r=1, world=2 2022-03-17T19:29:13.8845111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:13.8846155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:13.8860200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:15.2743660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:15.2744260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:15.8512518Z ok (2.847s) 2022-03-17T19:29:15.8512893Z 2022-03-17T19:29:15.8513388Z ---------------------------------------------------------------------- 2022-03-17T19:29:15.8513691Z Ran 2 tests in 4.404s 2022-03-17T19:29:15.8513834Z 2022-03-17T19:29:15.8513951Z OK (skipped=1) 2022-03-17T19:29:15.8514106Z 2022-03-17T19:29:15.8514212Z Generating XML reports... 2022-03-17T19:29:15.8557520Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20220317192911.xml 2022-03-17T19:29:16.0912276Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-03-17 19:29:16.090745] 2022-03-17T19:29:16.0912936Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:29:16.090850] 2022-03-17T19:29:16.9100616Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict 2022-03-17T19:29:16.9118491Z 2022-03-17T19:29:16.9118793Z Running tests... 2022-03-17T19:29:16.9119236Z ---------------------------------------------------------------------- 2022-03-17T19:29:16.9140017Z test_basic_save_and_load_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False (__main__.TestFSDPStateDict) 2022-03-17T19:29:18.4538274Z Tests that we can save a state_dict and load it into a blank model ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:29:18.4959306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31604 2022-03-17T19:29:18.5081591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31605 2022-03-17T19:29:19.3406541Z dist init r=1, world=2 2022-03-17T19:29:19.3409277Z dist init r=0, world=2 2022-03-17T19:29:19.3411243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:19.3414950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:19.3415989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:19.3514313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:20.7088638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:20.7089213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:20.7317052Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:20.7317870Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:20.7318888Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:20.7319648Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:21.0146345Z ok (4.102s) 2022-03-17T19:29:21.0166747Z test_basic_save_and_load_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True (__main__.TestFSDPStateDict) 2022-03-17T19:29:21.0307463Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31687 2022-03-17T19:29:21.0422693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31688 2022-03-17T19:29:21.8663577Z dist init r=0, world=2 2022-03-17T19:29:21.8668173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:21.8772099Z dist init r=1, world=2 2022-03-17T19:29:21.8777819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:21.8779398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:21.8872731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:23.2460814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:23.2461933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:23.2726122Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:23.2726897Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:23.2728042Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:23.2728833Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:23.5481422Z ok (2.533s) 2022-03-17T19:29:23.5501491Z test_basic_save_and_load_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False (__main__.TestFSDPStateDict) 2022-03-17T19:29:23.5640170Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31770 2022-03-17T19:29:23.5753393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31771 2022-03-17T19:29:24.3919599Z dist init r=0, world=2 2022-03-17T19:29:24.3924303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:24.4150624Z dist init r=1, world=2 2022-03-17T19:29:24.4156262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:24.4157449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:24.4229600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:25.7725526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:25.7726048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:25.7958129Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:25.7958834Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:25.7959881Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:25.7960624Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:26.0812757Z ok (2.533s) 2022-03-17T19:29:26.0833448Z test_basic_save_and_load_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True (__main__.TestFSDPStateDict) 2022-03-17T19:29:26.0974010Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31853 2022-03-17T19:29:26.1087539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31854 2022-03-17T19:29:26.9373524Z dist init r=0, world=2 2022-03-17T19:29:26.9378077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:26.9513115Z dist init r=1, world=2 2022-03-17T19:29:26.9518674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:26.9519730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:26.9582748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:28.3207668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:28.3208618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:28.3441226Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:28.3442807Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:28.3444844Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:28.3446265Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:28.6147419Z ok (2.533s) 2022-03-17T19:29:28.6296327Z test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31936 2022-03-17T19:29:28.6413361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31937 2022-03-17T19:29:29.4738388Z dist init r=1, world=2 2022-03-17T19:29:29.4742898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:29.4752073Z dist init r=0, world=2 2022-03-17T19:29:29.4757896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:29.4758813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:29.4846431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:30.8264132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:30.8264612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:30.8506143Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:30.8506884Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:30.8507947Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:30.8508718Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:31.1471279Z ok (2.532s) 2022-03-17T19:29:31.1627563Z test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32019 2022-03-17T19:29:31.1745992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32020 2022-03-17T19:29:31.9768528Z dist init r=0, world=2 2022-03-17T19:29:31.9772855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:32.0167955Z dist init r=1, world=2 2022-03-17T19:29:32.0173744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:32.0174536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:32.0179524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:33.3866445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:33.3866924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:33.4117504Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:33.4118266Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:33.4119276Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:33.4120046Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:33.6803283Z ok (2.533s) 2022-03-17T19:29:33.6818410Z test_save_and_load_after_forward_state_dict (__main__.TestFSDPStateDict) 2022-03-17T19:29:33.6958139Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32102 2022-03-17T19:29:33.7072342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32103 2022-03-17T19:29:34.5097456Z dist init r=0, world=2 2022-03-17T19:29:34.5102131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:34.5158752Z dist init r=1, world=2 2022-03-17T19:29:34.5164692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:34.5165497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:34.5205412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:35.8692998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:35.8693572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:36.6912039Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:36.6912905Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:36.6921529Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:36.6922587Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:37.1145762Z ok (3.434s) 2022-03-17T19:29:37.1161414Z test_state_dict_load_into_local_module (__main__.TestFSDPStateDict) 2022-03-17T19:29:37.1299139Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32189 2022-03-17T19:29:37.1412382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32190 2022-03-17T19:29:37.9914724Z dist init r=1, world=2 2022-03-17T19:29:37.9919330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:37.9919733Z dist init r=0, world=2 2022-03-17T19:29:37.9924583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:37.9925824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:38.0022147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:39.3680778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:39.3681305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:39.6949834Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:39.6950667Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:39.6951684Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:39.6952456Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:39.9475356Z ok (2.833s) 2022-03-17T19:29:39.9623574Z test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32276 2022-03-17T19:29:39.9737371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32277 2022-03-17T19:29:40.8353353Z dist init r=1, world=2 2022-03-17T19:29:40.8353650Z dist init r=0, world=2 2022-03-17T19:29:40.8358143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:40.8359027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:40.8359544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:40.8361000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:42.1953132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:42.1953674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:42.5225516Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:42.5226679Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:42.5227725Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:42.5228488Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:42.5320968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:29:42.5321463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:29:42.8802210Z ok (2.933s) 2022-03-17T19:29:42.8960890Z test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32363 2022-03-17T19:29:42.9082325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32364 2022-03-17T19:29:43.7637554Z dist init r=0, world=2 2022-03-17T19:29:43.7642569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:43.7800885Z dist init r=1, world=2 2022-03-17T19:29:43.7806919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:43.7808075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:43.7847208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:45.1294163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:45.1294719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:45.4578095Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:45.4578896Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:45.4579943Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:29:45.4580889Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:29:45.4676726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:29:45.4677217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:29:45.7144404Z ok (2.834s) 2022-03-17T19:29:45.7144788Z 2022-03-17T19:29:45.7145192Z ---------------------------------------------------------------------- 2022-03-17T19:29:45.7145529Z Ran 10 tests in 28.803s 2022-03-17T19:29:45.7145678Z 2022-03-17T19:29:45.7145769Z OK 2022-03-17T19:29:45.7145903Z 2022-03-17T19:29:45.7146035Z Generating XML reports... 2022-03-17T19:29:45.7201274Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220317192916.xml 2022-03-17T19:29:45.9574139Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-03-17 19:29:45.956900] 2022-03-17T19:29:45.9574875Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:29:45.957006] 2022-03-17T19:29:46.7803318Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params 2022-03-17T19:29:46.7826238Z 2022-03-17T19:29:46.7826813Z Running tests... 2022-03-17T19:29:46.7827250Z ---------------------------------------------------------------------- 2022-03-17T19:29:48.3184681Z test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:29:48.3603781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32487 2022-03-17T19:29:48.3728720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32488 2022-03-17T19:29:49.2003471Z dist init r=0, world=2 2022-03-17T19:29:49.2006929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:49.2444422Z dist init r=1, world=2 2022-03-17T19:29:49.2448928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:49.2450032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:49.2515170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:50.6287773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:50.6288258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:50.9357833Z Asserting FSDP instance is: FullyShardedDataParallel( 2022-03-17T19:29:50.9358261Z (_fsdp_wrapped_module): FlattenParamsWrapper( 2022-03-17T19:29:50.9358612Z (_fpw_module): Linear(in_features=2, out_features=1, bias=True) 2022-03-17T19:29:50.9358885Z ) 2022-03-17T19:29:50.9359094Z ) 2022-03-17T19:29:50.9359401Z ERROR: expected to be in states [] but current state is TrainingState_.BACKWARD_PRE 2022-03-17T19:29:50.9364135Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 200, in bad_backwards_hook 2022-03-17T19:29:50.9364660Z with model.summon_full_params(): 2022-03-17T19:29:50.9364969Z File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ 2022-03-17T19:29:50.9365272Z return next(self.gen) 2022-03-17T19:29:50.9365951Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1290, in summon_full_params 2022-03-17T19:29:50.9366312Z offload_to_cpu=offload_to_cpu, 2022-03-17T19:29:50.9366652Z File "/opt/conda/lib/python3.7/contextlib.py", line 427, in enter_context 2022-03-17T19:29:50.9367025Z result = _cm_type.__enter__(cm) 2022-03-17T19:29:50.9367318Z File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ 2022-03-17T19:29:50.9367706Z return next(self.gen) 2022-03-17T19:29:50.9368196Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1300, in summon_full_params 2022-03-17T19:29:50.9368647Z self._assert_state([TrainingState_.IDLE]) 2022-03-17T19:29:50.9369199Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1877, in _assert_state 2022-03-17T19:29:50.9369592Z traceback.print_stack() 2022-03-17T19:29:51.2799528Z ok (4.497s) 2022-03-17T19:29:51.2948593Z test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32574 2022-03-17T19:29:51.3064426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32575 2022-03-17T19:29:52.1148753Z dist init r=0, world=2 2022-03-17T19:29:52.1152533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:52.1492945Z dist init r=1, world=2 2022-03-17T19:29:52.1497368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:52.1498147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:52.1559574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:53.5267701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:53.5268259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:53.5500602Z Asserting FSDP instance is: FullyShardedDataParallel( 2022-03-17T19:29:53.5501293Z (_fsdp_wrapped_module): FlattenParamsWrapper( 2022-03-17T19:29:53.5501559Z (_fpw_module): MyModule() 2022-03-17T19:29:53.5501777Z ) 2022-03-17T19:29:53.5501984Z ) 2022-03-17T19:29:53.5502333Z ERROR: expected to be in states [] but current state is TrainingState_.FORWARD 2022-03-17T19:29:53.5518772Z File "", line 1, in 2022-03-17T19:29:53.5519213Z File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main 2022-03-17T19:29:53.5519504Z exitcode = _main(fd) 2022-03-17T19:29:53.5519908Z File "/opt/conda/lib/python3.7/multiprocessing/spawn.py", line 118, in _main 2022-03-17T19:29:53.5520169Z return self._bootstrap() 2022-03-17T19:29:53.5520495Z File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap 2022-03-17T19:29:53.5520803Z self.run() 2022-03-17T19:29:53.5521164Z File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 99, in run 2022-03-17T19:29:53.5523094Z self._target(*self._args, **self._kwargs) 2022-03-17T19:29:53.5523845Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_fsdp.py", line 399, in _run 2022-03-17T19:29:53.5524292Z self.run_test(test_name, pipe) 2022-03-17T19:29:53.5524896Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 601, in run_test 2022-03-17T19:29:53.5525575Z getattr(self, test_name)() 2022-03-17T19:29:53.5526084Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 486, in wrapper 2022-03-17T19:29:53.5526435Z fn() 2022-03-17T19:29:53.5526909Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 131, in wrapper 2022-03-17T19:29:53.5527310Z return func(*args, **kwargs) 2022-03-17T19:29:53.5527735Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 191, in test_cannot_summon_full_params_from_forward 2022-03-17T19:29:53.5528125Z model(model) 2022-03-17T19:29:53.5528595Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 2022-03-17T19:29:53.5528945Z return forward_call(*input, **kwargs) 2022-03-17T19:29:53.5529473Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1155, in forward 2022-03-17T19:29:53.5529894Z outputs = self.module(*args, **kwargs) 2022-03-17T19:29:53.5530408Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 2022-03-17T19:29:53.5530743Z return forward_call(*input, **kwargs) 2022-03-17T19:29:53.5531268Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/flatten_params_wrapper.py", line 454, in forward 2022-03-17T19:29:53.5531666Z return self.module(*inputs, **kwinputs) 2022-03-17T19:29:53.5532154Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl 2022-03-17T19:29:53.5532531Z return forward_call(*input, **kwargs) 2022-03-17T19:29:53.5532917Z File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 184, in forward 2022-03-17T19:29:53.5533315Z with fsdp_module.summon_full_params(): 2022-03-17T19:29:53.5533864Z File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ 2022-03-17T19:29:53.5534203Z return next(self.gen) 2022-03-17T19:29:53.5534753Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1290, in summon_full_params 2022-03-17T19:29:53.5535160Z offload_to_cpu=offload_to_cpu, 2022-03-17T19:29:53.5535504Z File "/opt/conda/lib/python3.7/contextlib.py", line 427, in enter_context 2022-03-17T19:29:53.5536140Z result = _cm_type.__enter__(cm) 2022-03-17T19:29:53.5536648Z File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ 2022-03-17T19:29:53.5536972Z return next(self.gen) 2022-03-17T19:29:53.5537583Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1300, in summon_full_params 2022-03-17T19:29:53.5538059Z self._assert_state([TrainingState_.IDLE]) 2022-03-17T19:29:53.5538578Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1877, in _assert_state 2022-03-17T19:29:53.5538990Z traceback.print_stack() 2022-03-17T19:29:53.8122194Z ok (2.532s) 2022-03-17T19:29:53.8280848Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32657 2022-03-17T19:29:53.8396008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32658 2022-03-17T19:29:54.6439719Z dist init r=0, world=2 2022-03-17T19:29:54.6443360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:54.6700160Z dist init r=1, world=2 2022-03-17T19:29:54.6705199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:54.6706190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:54.6749825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:56.0213990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:56.0214528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:56.3456003Z ok (2.533s) 2022-03-17T19:29:56.3615276Z test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32740 2022-03-17T19:29:56.3731715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32741 2022-03-17T19:29:57.1767777Z dist init r=0, world=2 2022-03-17T19:29:57.1768131Z dist init r=1, world=2 2022-03-17T19:29:57.1771300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:57.1771834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:57.1773074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:57.1773757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:58.5528156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:29:58.5528674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:29:58.5737747Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1267: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-03-17T19:29:58.5738825Z "offload_to_cpu and rank0_only=False will result in " 2022-03-17T19:29:58.5739846Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1267: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-03-17T19:29:58.5741060Z "offload_to_cpu and rank0_only=False will result in " 2022-03-17T19:29:58.8790097Z ok (2.533s) 2022-03-17T19:29:58.8961956Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32823 2022-03-17T19:29:58.9081494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32824 2022-03-17T19:29:59.7284782Z dist init r=0, world=2 2022-03-17T19:29:59.7287596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:29:59.7338131Z dist init r=1, world=2 2022-03-17T19:29:59.7343255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:29:59.7344195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:29:59.7390627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:01.0987403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:01.0987956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:01.4140168Z ok (2.535s) 2022-03-17T19:30:01.4297501Z test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32906 2022-03-17T19:30:01.4412190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32907 2022-03-17T19:30:02.2632010Z dist init r=1, world=2 2022-03-17T19:30:02.2635220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:02.2758824Z dist init r=0, world=2 2022-03-17T19:30:02.2763892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:02.2765237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:02.2839747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:03.6652545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:03.6653051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:03.9472252Z ok (2.533s) 2022-03-17T19:30:03.9628250Z test_params_count_and_value_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32989 2022-03-17T19:30:03.9742821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32990 2022-03-17T19:30:04.8000562Z dist init r=0, world=2 2022-03-17T19:30:04.8004041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:04.8075555Z dist init r=1, world=2 2022-03-17T19:30:04.8080675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:04.8081628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:04.8106986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:06.1739281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:06.1739794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:06.4800853Z ok (2.533s) 2022-03-17T19:30:06.4967797Z test_params_count_and_value_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33072 2022-03-17T19:30:06.5082784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33073 2022-03-17T19:30:07.3338594Z dist init r=0, world=2 2022-03-17T19:30:07.3341788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:07.3378746Z dist init r=1, world=2 2022-03-17T19:30:07.3383975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:07.3384774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:07.3445407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:08.7083383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:08.7084312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:09.0142124Z ok (2.534s) 2022-03-17T19:30:09.0301560Z test_params_count_and_value_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33155 2022-03-17T19:30:09.0417136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33156 2022-03-17T19:30:09.8477170Z dist init r=0, world=2 2022-03-17T19:30:09.8481016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:09.8868059Z dist init r=1, world=2 2022-03-17T19:30:09.8872705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:09.8873470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:09.8887495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:11.2574388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:11.2574927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:11.5477093Z ok (2.533s) 2022-03-17T19:30:11.5645650Z test_params_count_and_value_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33238 2022-03-17T19:30:11.5761310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33239 2022-03-17T19:30:12.3783116Z dist init r=0, world=2 2022-03-17T19:30:12.3787210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:12.4100454Z dist init r=1, world=2 2022-03-17T19:30:12.4105419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:12.4106418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:12.4194307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:13.7744362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:13.7744867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:14.0819341Z ok (2.534s) 2022-03-17T19:30:14.0967122Z test_raises_rank0_with_writeback (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33321 2022-03-17T19:30:14.1082691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33322 2022-03-17T19:30:14.9339694Z dist init r=0, world=2 2022-03-17T19:30:14.9342886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:14.9717365Z dist init r=1, world=2 2022-03-17T19:30:14.9722087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:14.9723388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:14.9749345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:16.3264411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:16.6142244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:16.6142611Z ok (2.532s) 2022-03-17T19:30:16.6305178Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33404 2022-03-17T19:30:16.6426876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33405 2022-03-17T19:30:17.4718111Z dist init r=0, world=2 2022-03-17T19:30:17.4721261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:17.5060492Z dist init r=1, world=2 2022-03-17T19:30:17.5065420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:17.5066283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:17.5127881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:18.8802507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:18.8803054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:19.4492102Z ok (2.835s) 2022-03-17T19:30:19.4651537Z test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33491 2022-03-17T19:30:19.4768350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33492 2022-03-17T19:30:20.3426465Z dist init r=0, world=2 2022-03-17T19:30:20.3429633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:20.3435262Z dist init r=1, world=2 2022-03-17T19:30:20.3440037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:20.3441123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:20.3532940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:21.7083714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:21.7084722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:22.0235398Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1267: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-03-17T19:30:22.0237285Z "offload_to_cpu and rank0_only=False will result in " 2022-03-17T19:30:22.0238978Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1267: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-03-17T19:30:22.0239730Z "offload_to_cpu and rank0_only=False will result in " 2022-03-17T19:30:22.2833206Z ok (2.834s) 2022-03-17T19:30:22.2994868Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33578 2022-03-17T19:30:22.3111143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33579 2022-03-17T19:30:23.1140780Z dist init r=1, world=2 2022-03-17T19:30:23.1144010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:23.1254007Z dist init r=0, world=2 2022-03-17T19:30:23.1258569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:23.1259469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:23.1348549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:24.4916871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:24.4917415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:25.1186373Z ok (2.835s) 2022-03-17T19:30:25.1352700Z test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33665 2022-03-17T19:30:25.1472467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33666 2022-03-17T19:30:25.9473476Z dist init r=0, world=2 2022-03-17T19:30:25.9477103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:25.9799399Z dist init r=1, world=2 2022-03-17T19:30:25.9803656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:25.9804436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:25.9883881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:27.3652643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:27.3653183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:27.9536797Z ok (2.835s) 2022-03-17T19:30:27.9694093Z test_summon_full_param_recursive_recurse_False_summon_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33752 2022-03-17T19:30:27.9811562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33753 2022-03-17T19:30:28.8226521Z dist init r=1, world=2 2022-03-17T19:30:28.8229609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:28.8387594Z dist init r=0, world=2 2022-03-17T19:30:28.8392371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:28.8393221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:28.8433596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:30.2037700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:30.2038211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:30.4870568Z ok (2.533s) 2022-03-17T19:30:30.5029215Z test_summon_full_param_recursive_recurse_False_summon_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33835 2022-03-17T19:30:30.5145109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33836 2022-03-17T19:30:31.3586526Z dist init r=0, world=2 2022-03-17T19:30:31.3589801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:31.3612749Z dist init r=1, world=2 2022-03-17T19:30:31.3617538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:31.3618736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:31.3693027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:32.7275639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:32.7276214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:33.0202680Z ok (2.533s) 2022-03-17T19:30:33.0358365Z test_summon_full_param_recursive_recurse_True_summon_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33918 2022-03-17T19:30:33.0475224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33919 2022-03-17T19:30:33.8659205Z dist init r=0, world=2 2022-03-17T19:30:33.8662806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:33.8730551Z dist init r=1, world=2 2022-03-17T19:30:33.8735504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:33.8736231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:33.8765994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:35.2328506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:35.2329036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:35.5532393Z ok (2.533s) 2022-03-17T19:30:35.5688334Z test_summon_full_param_recursive_recurse_True_summon_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34001 2022-03-17T19:30:35.5807583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34002 2022-03-17T19:30:36.4267584Z dist init r=0, world=2 2022-03-17T19:30:36.4271354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:36.4469952Z dist init r=1, world=2 2022-03-17T19:30:36.4475057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:36.4475589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:36.4476258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:37.8193864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:37.8194709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:38.0866313Z ok (2.533s) 2022-03-17T19:30:38.1019345Z test_summon_full_param_shard_value (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34084 2022-03-17T19:30:38.1136682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34085 2022-03-17T19:30:38.9357368Z dist init r=1, world=2 2022-03-17T19:30:38.9360951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:38.9772055Z dist init r=0, world=2 2022-03-17T19:30:38.9776759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:38.9777812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:38.9869280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:40.3589747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:40.3590263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:40.6192962Z ok (2.533s) 2022-03-17T19:30:40.6338542Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34167 2022-03-17T19:30:40.6454437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34168 2022-03-17T19:30:41.4792560Z dist init r=1, world=2 2022-03-17T19:30:41.4795733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:41.4894241Z dist init r=0, world=2 2022-03-17T19:30:41.4898641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:41.4899454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:41.4900189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:42.8512460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:42.8512944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:43.1512688Z ok (2.532s) 2022-03-17T19:30:43.1661651Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34250 2022-03-17T19:30:43.1779171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34251 2022-03-17T19:30:44.0218630Z dist init r=0, world=2 2022-03-17T19:30:44.0222329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:44.0483807Z dist init r=1, world=2 2022-03-17T19:30:44.0488604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:44.0489628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:44.0527925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:45.4074959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:45.4075811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:45.6835755Z ok (2.532s) 2022-03-17T19:30:45.6985709Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34333 2022-03-17T19:30:45.7101892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34334 2022-03-17T19:30:46.5176022Z dist init r=0, world=2 2022-03-17T19:30:46.5179343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:46.5531832Z dist init r=1, world=2 2022-03-17T19:30:46.5536314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:46.5537398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:46.5586025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:47.9132482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:47.9132997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:48.2159441Z ok (2.532s) 2022-03-17T19:30:48.2310795Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34416 2022-03-17T19:30:48.2428279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34417 2022-03-17T19:30:49.0479283Z dist init r=0, world=2 2022-03-17T19:30:49.0482624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:49.0497029Z dist init r=1, world=2 2022-03-17T19:30:49.0501497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:49.0503100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:49.0585677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:50.4205408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:50.4205964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:50.7485816Z ok (2.532s) 2022-03-17T19:30:50.7633074Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34499 2022-03-17T19:30:50.7749829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34500 2022-03-17T19:30:51.5873711Z dist init r=1, world=2 2022-03-17T19:30:51.5873991Z dist init r=0, world=2 2022-03-17T19:30:51.5876836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:51.5877351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:51.5878061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:51.5879005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:52.9653418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:52.9653993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:53.2807222Z ok (2.532s) 2022-03-17T19:30:53.2953359Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34582 2022-03-17T19:30:53.3069383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34583 2022-03-17T19:30:54.1129980Z dist init r=0, world=2 2022-03-17T19:30:54.1133287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:54.1137440Z dist init r=1, world=2 2022-03-17T19:30:54.1142545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:54.1143573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:54.1236737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:55.4870730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:55.4871240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:55.8127568Z ok (2.532s) 2022-03-17T19:30:55.8281272Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34665 2022-03-17T19:30:55.8399553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34666 2022-03-17T19:30:56.6758806Z dist init r=1, world=2 2022-03-17T19:30:56.6761997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:56.6885888Z dist init r=0, world=2 2022-03-17T19:30:56.6890568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:56.6891398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:56.6966086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:58.0553007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:30:58.0553528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:30:58.3456408Z ok (2.533s) 2022-03-17T19:30:58.3604332Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34748 2022-03-17T19:30:58.3721173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34749 2022-03-17T19:30:59.1661921Z dist init r=0, world=2 2022-03-17T19:30:59.1665431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:30:59.2238577Z dist init r=1, world=2 2022-03-17T19:30:59.2243534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:30:59.2244306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:30:59.2274150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:00.5875197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:00.5875654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:00.8778315Z ok (2.532s) 2022-03-17T19:31:00.8939081Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34831 2022-03-17T19:31:00.9056402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34832 2022-03-17T19:31:01.7455351Z dist init r=1, world=2 2022-03-17T19:31:01.7458489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:01.7544632Z dist init r=0, world=2 2022-03-17T19:31:01.7549006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:01.7549939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:01.7561154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:03.1126125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:03.1126667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:03.4115785Z ok (2.534s) 2022-03-17T19:31:03.4278163Z test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34914 2022-03-17T19:31:03.4396628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34915 2022-03-17T19:31:04.2667981Z dist init r=1, world=2 2022-03-17T19:31:04.2670748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:04.2685398Z dist init r=0, world=2 2022-03-17T19:31:04.2689987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:04.2690902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:04.2773797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:05.6585766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:05.6586316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:05.6829633Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1267: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-03-17T19:31:05.6830376Z "offload_to_cpu and rank0_only=False will result in " 2022-03-17T19:31:05.6864204Z /opt/conda/lib/python3.7/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1267: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True. 2022-03-17T19:31:05.6864940Z "offload_to_cpu and rank0_only=False will result in " 2022-03-17T19:31:05.9454473Z ok (2.534s) 2022-03-17T19:31:05.9611238Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34997 2022-03-17T19:31:05.9727905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34998 2022-03-17T19:31:06.7906496Z dist init r=1, world=2 2022-03-17T19:31:06.7909373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:06.8122750Z dist init r=0, world=2 2022-03-17T19:31:06.8127522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:06.8128649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:06.8214881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:08.1783752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:08.1784292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:08.4787626Z ok (2.533s) 2022-03-17T19:31:08.4943150Z test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35080 2022-03-17T19:31:08.5059231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35081 2022-03-17T19:31:09.3463549Z dist init r=0, world=2 2022-03-17T19:31:09.3466460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:09.3678204Z dist init r=1, world=2 2022-03-17T19:31:09.3682950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:09.3684042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:09.3771854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:10.7421775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:10.7422281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:11.0118841Z ok (2.533s) 2022-03-17T19:31:11.0273393Z test_summon_full_params_respects_reshard_after_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35163 2022-03-17T19:31:11.0392548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35164 2022-03-17T19:31:11.8360168Z dist init r=1, world=2 2022-03-17T19:31:11.8363442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:11.8512490Z dist init r=0, world=2 2022-03-17T19:31:11.8517209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:11.8518071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:11.8567809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:13.2167874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:13.2168388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:13.8466157Z ok (2.835s) 2022-03-17T19:31:13.8626078Z test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35246 2022-03-17T19:31:13.8747434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35247 2022-03-17T19:31:14.6832414Z dist init r=0, world=2 2022-03-17T19:31:14.6836046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:14.7169191Z dist init r=1, world=2 2022-03-17T19:31:14.7174445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:14.7175230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:14.7242815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:16.0937181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:16.0937735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:16.3815987Z ok (2.535s) 2022-03-17T19:31:16.3974994Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_modify_outer_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35329 2022-03-17T19:31:17.2112272Z dist init r=0, world=1 2022-03-17T19:31:17.2115987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:17.2116758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:18.5175805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:18.8027143Z ok (2.421s) 2022-03-17T19:31:18.8175967Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_modify_outer_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35371 2022-03-17T19:31:19.6261490Z dist init r=0, world=1 2022-03-17T19:31:19.6264567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:19.6265669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:20.9297461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:21.2234350Z ok (2.421s) 2022-03-17T19:31:21.2377788Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_modify_outer_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35413 2022-03-17T19:31:22.0465382Z dist init r=0, world=1 2022-03-17T19:31:22.0468970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:22.0469833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:23.3297300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:23.5430575Z ok (2.319s) 2022-03-17T19:31:23.5578912Z test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_modify_outer_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35455 2022-03-17T19:31:24.3710889Z dist init r=0, world=1 2022-03-17T19:31:24.3714193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:24.3715179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:25.6620424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:25.9632134Z ok (2.420s) 2022-03-17T19:31:25.9783016Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_modify_outer_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35497 2022-03-17T19:31:26.7878247Z dist init r=0, world=1 2022-03-17T19:31:26.7882048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:26.7882890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:28.0898748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:28.3835080Z ok (2.420s) 2022-03-17T19:31:28.3998196Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_modify_outer_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35539 2022-03-17T19:31:29.2073375Z dist init r=0, world=1 2022-03-17T19:31:29.2077611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:29.2078434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:30.5025609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:30.7046033Z ok (2.321s) 2022-03-17T19:31:30.7193432Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_modify_outer_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35581 2022-03-17T19:31:31.5236191Z dist init r=0, world=1 2022-03-17T19:31:31.5239686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:31.5240856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:32.8220615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:33.0242632Z ok (2.320s) 2022-03-17T19:31:33.0389447Z test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_modify_outer_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35623 2022-03-17T19:31:33.8510338Z dist init r=0, world=1 2022-03-17T19:31:33.8513760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:33.8514935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:35.1208967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:35.3441102Z ok (2.320s) 2022-03-17T19:31:35.3441382Z 2022-03-17T19:31:35.3441775Z ---------------------------------------------------------------------- 2022-03-17T19:31:35.3442106Z Ran 42 tests in 108.561s 2022-03-17T19:31:35.3444534Z 2022-03-17T19:31:35.3444948Z OK 2022-03-17T19:31:35.3445138Z 2022-03-17T19:31:35.3445285Z Generating XML reports... 2022-03-17T19:31:35.3523210Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220317192946.xml 2022-03-17T19:31:35.3534413Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220317192946.xml 2022-03-17T19:31:35.5888609Z Running distributed/fsdp/test_fsdp_traversal ... [2022-03-17 19:31:35.588364] 2022-03-17T19:31:35.5889320Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:31:35.588469] 2022-03-17T19:31:36.3856184Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal 2022-03-17T19:31:36.3872773Z 2022-03-17T19:31:36.3873073Z Running tests... 2022-03-17T19:31:36.3873500Z ---------------------------------------------------------------------- 2022-03-17T19:31:37.9136241Z test_fsdp_modules (__main__.TestTraversal) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:31:37.9559903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35702 2022-03-17T19:31:37.9687936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35703 2022-03-17T19:31:38.8117894Z dist init r=1, world=2 2022-03-17T19:31:38.8121590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:38.8246468Z dist init r=0, world=2 2022-03-17T19:31:38.8251044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:38.8252143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:38.8326007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:40.2172589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:40.2173088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:40.5754364Z ok (4.188s) 2022-03-17T19:31:40.5754532Z 2022-03-17T19:31:40.5755596Z ---------------------------------------------------------------------- 2022-03-17T19:31:40.5756265Z Ran 1 test in 4.188s 2022-03-17T19:31:40.5756572Z 2022-03-17T19:31:40.5757150Z OK 2022-03-17T19:31:40.5757418Z 2022-03-17T19:31:40.5757683Z Generating XML reports... 2022-03-17T19:31:40.5800538Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220317193136.xml 2022-03-17T19:31:40.8167991Z Running distributed/fsdp/test_fsdp_uneven ... [2022-03-17 19:31:40.816330] 2022-03-17T19:31:40.8168715Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:31:40.816435] 2022-03-17T19:31:41.6277912Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven 2022-03-17T19:31:41.6293933Z 2022-03-17T19:31:41.6294084Z Running tests... 2022-03-17T19:31:41.6294847Z ---------------------------------------------------------------------- 2022-03-17T19:31:41.6310623Z test_one_iteration (__main__.TestUnevenParamShard) 2022-03-17T19:31:43.1710685Z Test FSDP with uneven divide of parameter shards. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:31:43.2132860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35822 2022-03-17T19:31:43.2256378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35823 2022-03-17T19:31:44.0244043Z dist init r=0, world=2 2022-03-17T19:31:44.0247445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:44.0304461Z dist init r=1, world=2 2022-03-17T19:31:44.0309353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:44.0310686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:44.0351266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:45.4078275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:45.4078839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:46.0323182Z ok (4.403s) 2022-03-17T19:31:46.0323439Z 2022-03-17T19:31:46.0323843Z ---------------------------------------------------------------------- 2022-03-17T19:31:46.0324175Z Ran 1 test in 4.403s 2022-03-17T19:31:46.0324339Z 2022-03-17T19:31:46.0324339Z OK 2022-03-17T19:31:46.0324461Z 2022-03-17T19:31:46.0324599Z Generating XML reports... 2022-03-17T19:31:46.0366562Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220317193141.xml 2022-03-17T19:31:46.2665573Z Running distributed/fsdp/test_utils ... [2022-03-17 19:31:46.266084] 2022-03-17T19:31:46.2666116Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:31:46.266190] 2022-03-17T19:31:47.0843583Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils 2022-03-17T19:31:47.0859631Z 2022-03-17T19:31:47.0859887Z Running tests... 2022-03-17T19:31:47.0860330Z ---------------------------------------------------------------------- 2022-03-17T19:31:48.6474898Z test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (1.561s) 2022-03-17T19:31:48.6510638Z test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.004s) 2022-03-17T19:31:48.6543067Z test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.003s) 2022-03-17T19:31:48.6556251Z test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s) 2022-03-17T19:31:48.6556676Z 2022-03-17T19:31:48.6556971Z ---------------------------------------------------------------------- 2022-03-17T19:31:48.6557325Z Ran 4 tests in 1.570s 2022-03-17T19:31:48.6557472Z 2022-03-17T19:31:48.6557565Z OK 2022-03-17T19:31:48.6557701Z 2022-03-17T19:31:48.6557829Z Generating XML reports... 2022-03-17T19:31:48.6595235Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220317193147.xml 2022-03-17T19:31:48.8718307Z Running distributed/fsdp/test_wrap ... [2022-03-17 19:31:48.871311] 2022-03-17T19:31:48.8718964Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:31:48.871418] 2022-03-17T19:31:49.6848443Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap 2022-03-17T19:31:49.6869066Z 2022-03-17T19:31:49.6869510Z Running tests... 2022-03-17T19:31:49.6869992Z ---------------------------------------------------------------------- 2022-03-17T19:31:49.6878692Z test_always_wrap (__main__.TestAutoWrap) 2022-03-17T19:31:51.2277158Z Test to ensure that if `always_wrap_policy` is ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:31:51.2481615Z ok (1.561s) 2022-03-17T19:31:51.2488062Z test_auto_wrap_api (__main__.TestAutoWrap) 2022-03-17T19:31:51.2509147Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... ok (0.003s) 2022-03-17T19:31:51.2517654Z test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap) 2022-03-17T19:31:51.2529405Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... ok (0.002s) 2022-03-17T19:31:51.2536189Z test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap) 2022-03-17T19:31:51.2549111Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.2550532Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.2551776Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.2552397Z ok (0.002s) 2022-03-17T19:31:51.2558757Z test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap) 2022-03-17T19:31:51.2580952Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... ok (0.003s) 2022-03-17T19:31:51.2591135Z test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap) 2022-03-17T19:31:51.2603088Z Test to ensure force-leaf modules are not wrapped. ... ok (0.002s) 2022-03-17T19:31:51.2635709Z test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False) (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:51.2636561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:51.5546732Z ok (0.294s) 2022-03-17T19:31:51.5568875Z test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True) (__main__.TestAutoWrap) ... ok (0.002s) 2022-03-17T19:31:51.5598487Z test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False) (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:51.5599449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:51.5614183Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5615514Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5616765Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5618026Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5619276Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5653397Z ok (0.008s) 2022-03-17T19:31:51.5681260Z test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True) (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:51.5682145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:31:51.5765565Z ok (0.011s) 2022-03-17T19:31:51.5784551Z test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... ok (0.002s) 2022-03-17T19:31:51.5805381Z test_wrap_override_defaults (__main__.TestAutoWrap) ... ok (0.002s) 2022-03-17T19:31:51.5826443Z test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.002s) 2022-03-17T19:31:51.5846768Z test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.002s) 2022-03-17T19:31:51.5855041Z test_error_already_wrapped_nested_False_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-03-17T19:31:51.5871306Z Test that an error is raised if we attempt to wrap when submodules are ... [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5872665Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5875090Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5876897Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5878172Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.5879763Z [W python_variable.cpp:94] Warning: Deallocating Tensor that still has live PyObject references. This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it. Subsequent accesses to this tensor via the PyObject will now fail. (function concrete_decref_fn) 2022-03-17T19:31:51.6175723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35987 2022-03-17T19:31:51.6322284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35988 2022-03-17T19:31:52.4638757Z dist init r=1, world=2 2022-03-17T19:31:52.4641928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:52.4791728Z dist init r=0, world=2 2022-03-17T19:31:52.4796346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:52.4797418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:52.4846636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:53.8725049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:53.8725534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:54.1381277Z ok (2.553s) 2022-03-17T19:31:54.1390456Z test_error_already_wrapped_nested_False_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-03-17T19:31:54.1550538Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36070 2022-03-17T19:31:54.1684675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36071 2022-03-17T19:31:54.9778211Z dist init r=0, world=2 2022-03-17T19:31:54.9781248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:54.9853808Z dist init r=1, world=2 2022-03-17T19:31:54.9858870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:54.9859944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:54.9883964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:56.3595565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:56.3596089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:56.6742043Z ok (2.536s) 2022-03-17T19:31:56.6751549Z test_error_already_wrapped_nested_True_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) 2022-03-17T19:31:56.6910236Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36153 2022-03-17T19:31:56.7044996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36154 2022-03-17T19:31:57.5392131Z dist init r=0, world=2 2022-03-17T19:31:57.5395333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:31:57.5797558Z dist init r=1, world=2 2022-03-17T19:31:57.5802527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:57.5803068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:31:57.5803746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:31:58.9533599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:31:58.9534132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:31:59.2103901Z ok (2.536s) 2022-03-17T19:31:59.2113800Z test_error_already_wrapped_nested_True_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) 2022-03-17T19:31:59.2268658Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36236 2022-03-17T19:31:59.2402698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36237 2022-03-17T19:32:00.0695450Z dist init r=0, world=2 2022-03-17T19:32:00.0699066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:00.0764696Z dist init r=1, world=2 2022-03-17T19:32:00.0769545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:00.0770369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:00.0801871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:01.4640951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:01.4641495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:01.7465305Z ok (2.536s) 2022-03-17T19:32:01.7658816Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36319 2022-03-17T19:32:01.7803750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36320 2022-03-17T19:32:02.6181184Z dist init r=1, world=2 2022-03-17T19:32:02.6185051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:02.6476198Z dist init r=0, world=2 2022-03-17T19:32:02.6480948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:02.6482016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:02.6490018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:04.0194891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:04.0195412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:04.6871051Z ok (2.940s) 2022-03-17T19:32:04.7063806Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36406 2022-03-17T19:32:04.7200332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36407 2022-03-17T19:32:05.5335081Z dist init r=0, world=2 2022-03-17T19:32:05.5338436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:05.5627066Z dist init r=1, world=2 2022-03-17T19:32:05.5631678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:05.5632500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:05.5643819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:06.9440936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:06.9441448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:07.6266554Z ok (2.939s) 2022-03-17T19:32:07.6462125Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36493 2022-03-17T19:32:07.6597531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36494 2022-03-17T19:32:08.4611338Z dist init r=0, world=2 2022-03-17T19:32:08.4614704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:08.4864362Z dist init r=1, world=2 2022-03-17T19:32:08.4869045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:08.4869810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:08.4920080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:09.8690866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:09.8691396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:10.5663044Z ok (2.939s) 2022-03-17T19:32:10.5852819Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36580 2022-03-17T19:32:10.5988647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36581 2022-03-17T19:32:11.4216990Z dist init r=0, world=2 2022-03-17T19:32:11.4217318Z dist init r=1, world=2 2022-03-17T19:32:11.4220050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:11.4221688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:11.4222267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:11.4222948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:12.7881923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:12.7882858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:13.4053452Z ok (2.839s) 2022-03-17T19:32:13.4243014Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36667 2022-03-17T19:32:13.4377087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36668 2022-03-17T19:32:14.2446564Z dist init r=0, world=2 2022-03-17T19:32:14.2450135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:14.2757596Z dist init r=1, world=2 2022-03-17T19:32:14.2762713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:14.2763527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:14.2856530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:15.6569109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:15.6569648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:15.9437277Z ok (2.538s) 2022-03-17T19:32:15.9630730Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36750 2022-03-17T19:32:15.9766476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36751 2022-03-17T19:32:16.8339316Z dist init r=0, world=2 2022-03-17T19:32:16.8342602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:16.8344050Z dist init r=1, world=2 2022-03-17T19:32:16.8348922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:16.8350168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:16.8445914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:18.2122991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:18.2123562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:18.8831257Z ok (2.939s) 2022-03-17T19:32:18.9017297Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36837 2022-03-17T19:32:18.9151055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36838 2022-03-17T19:32:19.7713087Z dist init r=1, world=2 2022-03-17T19:32:19.7716465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:19.7718623Z dist init r=0, world=2 2022-03-17T19:32:19.7723369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:19.7724459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:19.7819423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:21.1419546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:21.1420095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:21.4208956Z ok (2.538s) 2022-03-17T19:32:21.4396657Z test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36920 2022-03-17T19:32:21.4530066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36921 2022-03-17T19:32:22.2599334Z dist init r=1, world=2 2022-03-17T19:32:22.2602673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:22.2732813Z dist init r=0, world=2 2022-03-17T19:32:22.2738085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:22.2738892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:22.2805989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:23.6530082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:23.6530589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:24.3594975Z ok (2.938s) 2022-03-17T19:32:24.3595179Z 2022-03-17T19:32:24.3595539Z ---------------------------------------------------------------------- 2022-03-17T19:32:24.3595884Z Ran 26 tests in 34.673s 2022-03-17T19:32:24.3598476Z 2022-03-17T19:32:24.3598858Z OK 2022-03-17T19:32:24.3599057Z 2022-03-17T19:32:24.3599199Z Generating XML reports... 2022-03-17T19:32:24.3657441Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220317193149.xml 2022-03-17T19:32:24.3673684Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220317193149.xml 2022-03-17T19:32:24.6170789Z Running distributed/nn/jit/test_instantiator ... [2022-03-17 19:32:24.616563] 2022-03-17T19:32:24.6171782Z Executing ['/opt/conda/bin/python', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:32:24.616664] 2022-03-17T19:32:25.4256953Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator 2022-03-17T19:32:25.4272741Z 2022-03-17T19:32:25.4273264Z Running tests... 2022-03-17T19:32:25.4273830Z ---------------------------------------------------------------------- 2022-03-17T19:32:26.9749869Z test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (1.547s) 2022-03-17T19:32:26.9770289Z test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s) 2022-03-17T19:32:26.9929070Z test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.016s) 2022-03-17T19:32:26.9929357Z 2022-03-17T19:32:26.9929926Z ---------------------------------------------------------------------- 2022-03-17T19:32:26.9930289Z Ran 3 tests in 1.566s 2022-03-17T19:32:26.9931567Z 2022-03-17T19:32:26.9933436Z OK 2022-03-17T19:32:26.9933736Z 2022-03-17T19:32:26.9933874Z Generating XML reports... 2022-03-17T19:32:26.9965781Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220317193225.xml 2022-03-17T19:32:27.1987437Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-03-17 19:32:27.198254] 2022-03-17T19:32:27.1988203Z Executing ['/opt/conda/bin/python', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:32:27.198357] 2022-03-17T19:32:28.2422639Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer 2022-03-17T19:32:28.2453447Z 2022-03-17T19:32:28.2453899Z Running tests... 2022-03-17T19:32:28.2454510Z ---------------------------------------------------------------------- 2022-03-17T19:32:28.2476026Z test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:29.7736811Z Check that ZeroRedundancyOptimizer properly handles adding a new ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:32:29.7884879Z skip (1.543s) 2022-03-17T19:32:29.7905081Z test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:29.8163420Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37080 2022-03-17T19:32:29.8285293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37081 2022-03-17T19:32:30.8826483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:30.8828818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:30.9022771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:30.9026907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:30.9027718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:30.9032410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:33.9375536Z ok (4.149s) 2022-03-17T19:32:33.9390886Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:33.9531432Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37168 2022-03-17T19:32:33.9648569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37169 2022-03-17T19:32:35.0026484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:35.0029136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:35.0082463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:35.0086224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:35.0087049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:35.0132195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:36.4115210Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:36.4117924Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:36.7171872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:36.7172439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:36.7595297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:36.7595810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:37.0715833Z ok (3.134s) 2022-03-17T19:32:37.0730849Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:37.0869710Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37285 2022-03-17T19:32:37.0984537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37286 2022-03-17T19:32:38.1461577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:38.1463205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:38.1463703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:38.1467262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:38.1468505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:38.1567035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:39.5630168Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:39.5632722Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:39.8661318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:39.8661842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:39.9086100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:39.9086548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:40.2056988Z ok (3.134s) 2022-03-17T19:32:40.2071868Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:40.2215941Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37402 2022-03-17T19:32:40.2332497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37403 2022-03-17T19:32:41.2929590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:41.2931971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:41.3362369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:41.3366040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:41.3367221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:41.3439403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:42.7603997Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:42.7606535Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:43.0668060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:43.0668602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:43.1128153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:43.1128890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:43.4402712Z ok (3.234s) 2022-03-17T19:32:43.4417596Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:43.4554966Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37519 2022-03-17T19:32:43.4669577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37520 2022-03-17T19:32:44.4878100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:44.4879975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:44.4974317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:44.4978960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:44.4981030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:44.4983633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:45.8903387Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:45.8905845Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:46.1988645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:46.1989175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:46.2459026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:46.2459541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:46.5737243Z ok (3.133s) 2022-03-17T19:32:46.5752213Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:46.5895422Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37636 2022-03-17T19:32:46.6011112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37637 2022-03-17T19:32:47.6401675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:47.6403997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:47.6455378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:47.6459326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:47.6460744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:47.6507379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:49.0425759Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:49.0430417Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:49.3439192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:49.3439725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:49.3811833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:49.3812385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:49.7081526Z ok (3.134s) 2022-03-17T19:32:49.7095927Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:49.7236286Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37753 2022-03-17T19:32:49.7355949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37754 2022-03-17T19:32:50.7723370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:50.7725402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:50.7913638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:50.7917344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:50.7918688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:50.7929992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:52.2039024Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:52.2041736Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:52.5112770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:52.5113320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:52.5501702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:52.5502193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:52.8423464Z ok (3.134s) 2022-03-17T19:32:52.8438870Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:52.8575273Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37870 2022-03-17T19:32:52.8690588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37871 2022-03-17T19:32:53.9555201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:53.9557610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:53.9585614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:53.9589379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:53.9590636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:53.9660517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:55.3591926Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:55.3596026Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:55.6657291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:55.6657819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:55.7081419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:55.7082219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:56.0763440Z ok (3.234s) 2022-03-17T19:32:56.0779125Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:56.0922809Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37987 2022-03-17T19:32:56.1043819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37988 2022-03-17T19:32:57.1269351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:32:57.1272512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:32:57.1277399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:32:57.1281309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:32:57.1282877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:57.1375557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:32:58.5609913Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:58.5612345Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:32:58.8670726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:58.8671226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:58.9109932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:58.9110425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:32:59.2127018Z ok (3.136s) 2022-03-17T19:32:59.2142455Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:32:59.2282625Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38104 2022-03-17T19:32:59.2398691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38105 2022-03-17T19:33:00.3064086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:00.3066231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:00.3127703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:00.3132350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:00.3133779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:00.3169679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:01.7241760Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:01.7245434Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:02.0257034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:02.0257574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:02.0689614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:02.0690159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:02.3471353Z ok (3.134s) 2022-03-17T19:33:02.3486607Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:02.3631631Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38221 2022-03-17T19:33:02.3748671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38222 2022-03-17T19:33:03.4149721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:03.4152390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:03.4235936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:03.4239408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:03.4240326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:03.4255252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:04.8412484Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:04.8414775Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:05.1423955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:05.1424451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:05.1869946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:05.1870438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:05.4818155Z ok (3.135s) 2022-03-17T19:33:05.4833454Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:05.4970455Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38338 2022-03-17T19:33:05.5084923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38339 2022-03-17T19:33:06.5858163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:06.5861608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:06.6210358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:06.6214609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:06.6215408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:06.6268279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:08.0454392Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:08.0456564Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:08.3526799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:08.3527349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:08.4014436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:08.4015109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:08.7156728Z ok (3.234s) 2022-03-17T19:33:08.7171050Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:08.7313777Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38455 2022-03-17T19:33:08.7428840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38456 2022-03-17T19:33:09.7811024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:09.7813514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:09.7853068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:09.7858068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:09.7859533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:09.7917299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:11.1850718Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:11.1854024Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:11.4934623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:11.4935216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:11.5432345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:11.8509957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:11.8510377Z ok (3.135s) 2022-03-17T19:33:11.8524788Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:11.8664203Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38572 2022-03-17T19:33:11.8778691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38573 2022-03-17T19:33:12.9435657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:12.9437826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:12.9638508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:12.9642551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:12.9643654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:12.9743148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:14.3709112Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:14.3711717Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:14.6701653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:14.6702143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:14.7119368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:14.7119862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:15.0849583Z ok (3.234s) 2022-03-17T19:33:15.0864491Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:15.1008081Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38689 2022-03-17T19:33:15.1125880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38690 2022-03-17T19:33:16.1792598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:16.1793123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:16.1795663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:16.1796198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:16.1796972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:16.1797646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:17.5935415Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:17.5937618Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:17.8938174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:17.9361583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:17.9362057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:17.9362516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:18.2194940Z ok (3.134s) 2022-03-17T19:33:18.2209775Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:18.2347709Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38806 2022-03-17T19:33:18.2467135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38807 2022-03-17T19:33:19.3172603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:19.3174847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:19.3179627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:19.3184297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:19.3185020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:19.3277958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:20.7236048Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:20.7238446Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:21.0301694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:21.0302703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:21.0782482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:21.0783016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:21.4537452Z ok (3.234s) 2022-03-17T19:33:21.4552902Z test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:21.4694990Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38923 2022-03-17T19:33:21.5216512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38924 2022-03-17T19:33:22.5598744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:22.5601331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:22.6262365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:22.6266563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:22.6267666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:22.6311343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:24.0361198Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:24.0363999Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer instead of since `overlap_with_ddp=True` 2022-03-17T19:33:24.3425071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:24.3425597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:24.3902314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:24.3902817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:33:24.7287111Z ok (3.275s) 2022-03-17T19:33:24.7331942Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:24.7471734Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39040 2022-03-17T19:33:24.7590023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39041 2022-03-17T19:33:25.8195176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:25.8197452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:25.8459257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:25.8463704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:25.8464522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:25.8502981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:27.5274283Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:27.5276090Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:27.7905244Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.7906310Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8094153Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8102785Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8302009Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8302741Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8492298Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8501049Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8699641Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8700686Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8890469Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.8899133Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.9097846Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.9098602Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.9484344Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:27.9485109Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:28.2663497Z ok (3.538s) 2022-03-17T19:33:28.2707337Z test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:28.2844885Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39127 2022-03-17T19:33:28.2960356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39128 2022-03-17T19:33:29.3688021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:29.3691801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:29.3797432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:29.3800638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:29.3801672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:29.3895492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:31.0522150Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:31.0554870Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:31.3176925Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3177701Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3374243Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3375313Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3573867Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3574603Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3770136Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3772998Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3972982Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.3973712Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.4169587Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.4172107Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.4371859Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.4372832Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.4766599Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.4768096Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:31.8037049Z ok (3.537s) 2022-03-17T19:33:31.8084026Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:31.8227416Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39214 2022-03-17T19:33:31.8343800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39215 2022-03-17T19:33:32.8683093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:32.8686114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:32.8805960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:32.8809496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:32.8810654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:32.8890223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:34.5550153Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:34.5551779Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:34.8058976Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8059750Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8245067Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8249191Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8439750Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8440494Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8627066Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8630159Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8821272Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.8822030Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.9008458Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.9011614Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.9202720Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.9203459Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.9581673Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:34.9583681Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:35.2419125Z ok (3.438s) 2022-03-17T19:33:35.2464530Z test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:35.2602005Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39301 2022-03-17T19:33:35.2715566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39302 2022-03-17T19:33:36.3160383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:36.3163051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:36.3249160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:36.3252966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:36.3253770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:36.3265700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:37.9966332Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:37.9996275Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:38.2582424Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.2583353Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.2776601Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.2781403Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.2979548Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.2980682Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.3174828Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.3180344Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.3378391Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.3379398Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.3573525Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.3579142Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.3773966Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.3775026Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.4168193Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.4171181Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:38.7791968Z ok (3.537s) 2022-03-17T19:33:38.7836154Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:38.7974127Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39388 2022-03-17T19:33:38.8089763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39389 2022-03-17T19:33:39.7919197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:39.7921922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:39.8035987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:39.8040117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:39.8041127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:39.8126440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:41.4772643Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:41.4820191Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:41.7228210Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7228968Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7399931Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7411463Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7595646Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7596362Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7769342Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7778418Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7963535Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.7964231Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.8136641Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.8146946Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.8330930Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.8331775Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.8532415Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:41.8534736Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:42.2163941Z ok (3.437s) 2022-03-17T19:33:42.2208680Z test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:42.2354311Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39475 2022-03-17T19:33:42.2471300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39476 2022-03-17T19:33:43.3137827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:43.3140826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:43.3221072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:43.3224964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:43.3225741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:43.3243611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:44.9922484Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:44.9953128Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:33:45.2403692Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.2404368Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.2584160Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.2590321Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.2776500Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.2777181Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.2957300Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.2963235Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.3149675Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.3150335Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.3330441Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.3336421Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.3523246Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.3523914Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.3727529Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.3729759Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled 2022-03-17T19:33:45.6546594Z ok (3.438s) 2022-03-17T19:33:45.6561433Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:45.6700543Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39562 2022-03-17T19:33:45.6814944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39563 2022-03-17T19:33:46.7235922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:46.7238990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:46.7399498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:46.7403188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:46.7404404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:46.7443118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:49.3896070Z ok (3.735s) 2022-03-17T19:33:49.3926417Z test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:49.4066197Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39650 2022-03-17T19:33:49.4180978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39651 2022-03-17T19:33:50.4745402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:50.4747894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:50.4774678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:50.4778880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:50.4779993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:50.4851450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:53.5270295Z ok (4.137s) 2022-03-17T19:33:53.5303751Z test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:53.5445720Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39738 2022-03-17T19:33:53.5562860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39739 2022-03-17T19:33:54.6042173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:54.6042791Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-03-17T19:33:54.6177689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:54.6179506Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4 2022-03-17T19:33:54.7598678Z ok (1.233s) 2022-03-17T19:33:54.7609645Z test_sharding (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:54.7613178Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip (0.001s) 2022-03-17T19:33:54.7632696Z test_step (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:54.7773258Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39810 2022-03-17T19:33:54.7890060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39811 2022-03-17T19:33:55.8570746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:55.8573105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:55.8612468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:55.8616308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:55.8617346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:55.8676108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:57.7958127Z ok (3.034s) 2022-03-17T19:33:57.7981640Z test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:33:57.8125573Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39897 2022-03-17T19:33:57.8241922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39898 2022-03-17T19:33:58.8775477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:33:58.8777746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:33:58.8905079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:33:58.8905865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:33:58.8907302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:33:58.8982978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:00.8324158Z ok (3.036s) 2022-03-17T19:34:00.8329541Z test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:34:00.8470531Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39984 2022-03-17T19:34:00.8587055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39985 2022-03-17T19:34:01.9582039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:01.9748723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:01.9898105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:01.9898587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:34:01.9899370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:01.9900089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:02.0016703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:34:02.0017230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:34:02.2627225Z ok (1.430s) 2022-03-17T19:34:02.2632321Z test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:34:02.2777749Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40066 2022-03-17T19:34:02.2898665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40067 2022-03-17T19:34:03.3529275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:03.3538176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:03.3573150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:03.3582142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:34:03.3583170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:03.3641316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:34:05.9534752Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:34:05.9535280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:34:06.4004024Z ok (4.138s) 2022-03-17T19:34:06.4011418Z test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:34:06.4150714Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40154 2022-03-17T19:34:06.4268634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40155 2022-03-17T19:34:07.4746175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:07.4787427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:07.6303745Z skip (1.230s) 2022-03-17T19:34:07.6311312Z test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed) 2022-03-17T19:34:07.6449640Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40226 2022-03-17T19:34:07.6565116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40227 2022-03-17T19:34:08.7185235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:08.7250037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:34:08.8599695Z skip (1.229s) 2022-03-17T19:34:08.8622858Z test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:34:08.8760356Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40298 2022-03-17T19:34:09.8666999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:09.8669306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:09.8670388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:34:09.9793383Z ok (1.119s) 2022-03-17T19:34:09.9808928Z test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:34:09.9948002Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40335 2022-03-17T19:34:11.0312549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:11.0315284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:11.0316477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:34:12.6005212Z ok (2.621s) 2022-03-17T19:34:12.6015911Z test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:34:12.6159213Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40379 2022-03-17T19:34:13.6310042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:13.6312704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:13.6313883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:34:13.8190118Z ok (1.218s) 2022-03-17T19:34:13.8219750Z test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:34:13.8357668Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40416 2022-03-17T19:34:14.8266161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:14.8268252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:14.8269058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:34:16.3413728Z ok (2.522s) 2022-03-17T19:34:16.3426264Z test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:34:16.3563262Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40460 2022-03-17T19:34:17.3554739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:17.3557173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:17.3558036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:34:18.8617946Z ok (2.520s) 2022-03-17T19:34:18.8631231Z test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:34:18.8771701Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40504 2022-03-17T19:34:19.9445375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:19.9448607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:19.9449401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:34:21.4826998Z ok (2.621s) 2022-03-17T19:34:21.4837453Z test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:34:21.4982958Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40548 2022-03-17T19:34:22.5111286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:22.5113765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:22.5114841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:34:24.0037927Z ok (2.521s) 2022-03-17T19:34:24.0048993Z test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank) 2022-03-17T19:34:24.0188966Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40592 2022-03-17T19:34:25.0806404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:34:25.0809155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:34:25.0810104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:34:25.2221628Z ok (1.218s) 2022-03-17T19:34:25.2221849Z 2022-03-17T19:34:25.2222299Z ---------------------------------------------------------------------- 2022-03-17T19:34:25.2222665Z Ran 42 tests in 116.977s 2022-03-17T19:34:25.2223706Z 2022-03-17T19:34:25.2223842Z OK (skipped=4) 2022-03-17T19:34:25.2224013Z 2022-03-17T19:34:25.2230193Z Generating XML reports... 2022-03-17T19:34:25.2323823Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220317193228.xml 2022-03-17T19:34:25.2336286Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220317193228.xml 2022-03-17T19:34:25.4692794Z Running distributed/pipeline/sync/skip/test_api ... [2022-03-17 19:34:25.468774] 2022-03-17T19:34:25.4693422Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-03-17 19:34:25.468880] 2022-03-17T19:34:26.4145253Z ============================= test session starts ============================== 2022-03-17T19:34:26.4145865Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:26.4167721Z cachedir: .pytest_cache 2022-03-17T19:34:26.4168685Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:26.4169358Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:26.4169688Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:26.4170108Z plugins: hypothesis-4.53.2 2022-03-17T19:34:26.4324998Z collecting ...  2022-03-17T19:34:26.4325367Z collected 3 items  2022-03-17T19:34:26.4329860Z 2022-03-17T19:34:26.4364529Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%] 2022-03-17T19:34:26.4381135Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED [ 66%] 2022-03-17T19:34:26.4417141Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED [100%] 2022-03-17T19:34:26.4419499Z 2022-03-17T19:34:26.4419972Z ============================== 3 passed in 0.03s =============================== 2022-03-17T19:34:26.5447509Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-03-17 19:34:26.544270] 2022-03-17T19:34:26.5448103Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-03-17 19:34:26.544379] 2022-03-17T19:34:27.5016700Z ============================= test session starts ============================== 2022-03-17T19:34:27.5017283Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:27.5038961Z cachedir: .pytest_cache 2022-03-17T19:34:27.5039853Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:27.5040279Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:27.5040637Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:27.5041262Z plugins: hypothesis-4.53.2 2022-03-17T19:34:27.5339356Z collecting ...  2022-03-17T19:34:27.5339788Z collected 13 items  2022-03-17T19:34:27.5345065Z 2022-03-17T19:34:29.6401905Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] PASSED [ 7%] 2022-03-17T19:34:31.1771110Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] PASSED [ 15%] 2022-03-17T19:34:31.2263922Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] PASSED [ 23%] 2022-03-17T19:34:31.2416613Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%] 2022-03-17T19:34:31.3073523Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] PASSED [ 38%] 2022-03-17T19:34:31.3659384Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] PASSED [ 46%] 2022-03-17T19:34:31.4224925Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] PASSED [ 53%] 2022-03-17T19:34:31.4378728Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%] 2022-03-17T19:34:31.4941440Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] PASSED [ 69%] 2022-03-17T19:34:31.5481582Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] PASSED [ 76%] 2022-03-17T19:34:31.6002001Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] PASSED [ 84%] 2022-03-17T19:34:31.6154877Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%] 2022-03-17T19:34:31.6404560Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED [100%] 2022-03-17T19:34:31.6408239Z 2022-03-17T19:34:31.6408699Z =========================== short test summary info ============================ 2022-03-17T19:34:31.6409161Z SKIPPED [3] distributed/pipeline/sync/skip/test_gpipe.py:24: at least 3 cuda devices required 2022-03-17T19:34:31.6410245Z ======================== 10 passed, 3 skipped in 4.14s ========================= 2022-03-17T19:34:32.1759056Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-03-17 19:34:32.175394] 2022-03-17T19:34:32.1759707Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-03-17 19:34:32.175504] 2022-03-17T19:34:33.1597249Z ============================= test session starts ============================== 2022-03-17T19:34:33.1597814Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:33.1619557Z cachedir: .pytest_cache 2022-03-17T19:34:33.1620682Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:33.1621195Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:33.1621522Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:33.1621949Z plugins: hypothesis-4.53.2 2022-03-17T19:34:33.1798514Z collecting ...  2022-03-17T19:34:33.1798903Z collected 6 items  2022-03-17T19:34:33.1804214Z 2022-03-17T19:34:33.1840254Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%] 2022-03-17T19:34:33.1867515Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%] 2022-03-17T19:34:33.1886331Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%] 2022-03-17T19:34:33.1905871Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%] 2022-03-17T19:34:33.1925854Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%] 2022-03-17T19:34:33.1949585Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%] 2022-03-17T19:34:33.1951321Z 2022-03-17T19:34:33.1951625Z ============================== 6 passed in 0.04s =============================== 2022-03-17T19:34:33.2983581Z Running distributed/pipeline/sync/skip/test_leak ... [2022-03-17 19:34:33.297844] 2022-03-17T19:34:33.2984236Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-03-17 19:34:33.297949] 2022-03-17T19:34:34.2654928Z ============================= test session starts ============================== 2022-03-17T19:34:34.2655495Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:34.2677472Z cachedir: .pytest_cache 2022-03-17T19:34:34.2678282Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:34.2678669Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:34.2679015Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:34.2679401Z plugins: hypothesis-4.53.2 2022-03-17T19:34:34.2856073Z collecting ...  2022-03-17T19:34:34.2856500Z collected 8 items  2022-03-17T19:34:34.2861254Z 2022-03-17T19:34:34.3733149Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%] 2022-03-17T19:34:34.3918651Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%] 2022-03-17T19:34:34.4128811Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%] 2022-03-17T19:34:34.4312228Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%] 2022-03-17T19:34:34.4506881Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%] 2022-03-17T19:34:34.4692190Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%] 2022-03-17T19:34:34.4847969Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%] 2022-03-17T19:34:34.5005792Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%] 2022-03-17T19:34:34.5007155Z 2022-03-17T19:34:34.5007523Z ============================== 8 passed in 0.24s =============================== 2022-03-17T19:34:34.6121846Z Running distributed/pipeline/sync/skip/test_portal ... [2022-03-17 19:34:34.611666] 2022-03-17T19:34:34.6122503Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-03-17 19:34:34.611774] 2022-03-17T19:34:35.5704308Z ============================= test session starts ============================== 2022-03-17T19:34:35.5704892Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:35.5726613Z cachedir: .pytest_cache 2022-03-17T19:34:35.5727325Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:35.5727756Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:35.5728063Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:35.5728431Z plugins: hypothesis-4.53.2 2022-03-17T19:34:35.6142779Z collecting ...  2022-03-17T19:34:35.6143192Z collected 10 items  2022-03-17T19:34:35.6147931Z 2022-03-17T19:34:36.8595732Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%] 2022-03-17T19:34:36.8623887Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED [ 20%] 2022-03-17T19:34:36.8642528Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%] 2022-03-17T19:34:36.8657598Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED [ 40%] 2022-03-17T19:34:36.8674212Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%] 2022-03-17T19:34:36.8690127Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%] 2022-03-17T19:34:36.8706516Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%] 2022-03-17T19:34:36.8722812Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%] 2022-03-17T19:34:36.8739100Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%] 2022-03-17T19:34:36.8760125Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%] 2022-03-17T19:34:36.8762065Z 2022-03-17T19:34:36.8762513Z ============================== 10 passed in 1.31s ============================== 2022-03-17T19:34:37.0541892Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-03-17 19:34:37.053690] 2022-03-17T19:34:37.0542513Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-03-17 19:34:37.053798] 2022-03-17T19:34:38.0234677Z ============================= test session starts ============================== 2022-03-17T19:34:38.0235229Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:38.0256642Z cachedir: .pytest_cache 2022-03-17T19:34:38.0257422Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:38.0258109Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:38.0258438Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:38.0258829Z plugins: hypothesis-4.53.2 2022-03-17T19:34:38.0427465Z collecting ...  2022-03-17T19:34:38.0427871Z collected 7 items  2022-03-17T19:34:38.0432532Z 2022-03-17T19:34:38.0482566Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED [ 14%] 2022-03-17T19:34:38.0504130Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED [ 28%] 2022-03-17T19:34:38.0526329Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%] 2022-03-17T19:34:38.0546109Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%] 2022-03-17T19:34:38.0567171Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%] 2022-03-17T19:34:38.0586841Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%] 2022-03-17T19:34:38.0609868Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%] 2022-03-17T19:34:38.0611392Z 2022-03-17T19:34:38.0611820Z ============================== 7 passed in 0.04s =============================== 2022-03-17T19:34:38.1651777Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-03-17 19:34:38.164685] 2022-03-17T19:34:38.1652464Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-03-17 19:34:38.164795] 2022-03-17T19:34:39.1160345Z ============================= test session starts ============================== 2022-03-17T19:34:39.1160914Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:39.1182554Z cachedir: .pytest_cache 2022-03-17T19:34:39.1183573Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:39.1184007Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:39.1184340Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:39.1184725Z plugins: hypothesis-4.53.2 2022-03-17T19:34:39.1504521Z collecting ...  2022-03-17T19:34:39.1504932Z collected 6 items  2022-03-17T19:34:39.1509325Z 2022-03-17T19:34:39.1546598Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%] 2022-03-17T19:34:40.4058631Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%] 2022-03-17T19:34:40.4075122Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%] 2022-03-17T19:34:40.4089678Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%] 2022-03-17T19:34:40.4104779Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%] 2022-03-17T19:34:40.4124562Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%] 2022-03-17T19:34:40.4125711Z 2022-03-17T19:34:40.4126030Z ============================== 6 passed in 1.30s =============================== 2022-03-17T19:34:40.5882579Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-03-17 19:34:40.587782] 2022-03-17T19:34:40.5883252Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-03-17 19:34:40.587887] 2022-03-17T19:34:41.5219606Z ============================= test session starts ============================== 2022-03-17T19:34:41.5220162Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:41.5242537Z cachedir: .pytest_cache 2022-03-17T19:34:41.5243504Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:41.5243939Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:41.5244251Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:41.5244643Z plugins: hypothesis-4.53.2 2022-03-17T19:34:41.5449341Z collecting ...  2022-03-17T19:34:41.5449815Z collected 9 items  2022-03-17T19:34:41.5454571Z 2022-03-17T19:34:41.5492456Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%] 2022-03-17T19:34:41.5512206Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%] 2022-03-17T19:34:41.5531035Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%] 2022-03-17T19:34:41.5552025Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%] 2022-03-17T19:34:41.5572354Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%] 2022-03-17T19:34:41.5594994Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%] 2022-03-17T19:34:41.5613997Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%] 2022-03-17T19:34:41.5634850Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%] 2022-03-17T19:34:41.5659511Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%] 2022-03-17T19:34:41.5660824Z 2022-03-17T19:34:41.5661182Z ============================== 9 passed in 0.04s =============================== 2022-03-17T19:34:41.6704222Z Running distributed/pipeline/sync/test_balance ... [2022-03-17 19:34:41.669923] 2022-03-17T19:34:41.6704871Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-03-17 19:34:41.670066] 2022-03-17T19:34:42.6331463Z ============================= test session starts ============================== 2022-03-17T19:34:42.6332009Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:42.6353666Z cachedir: .pytest_cache 2022-03-17T19:34:42.6354365Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:42.6354785Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:42.6355126Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:42.6355516Z plugins: hypothesis-4.53.2 2022-03-17T19:34:42.6738079Z collecting ...  2022-03-17T19:34:42.6738494Z collected 18 items  2022-03-17T19:34:42.6743427Z 2022-03-17T19:34:42.6777281Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED [ 5%] 2022-03-17T19:34:42.6794525Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%] 2022-03-17T19:34:42.6812103Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%] 2022-03-17T19:34:42.6829370Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%] 2022-03-17T19:34:42.6841580Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%] 2022-03-17T19:34:42.6852712Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%] 2022-03-17T19:34:43.6903335Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%] 2022-03-17T19:34:44.9646305Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%] 2022-03-17T19:34:45.2592383Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%] 2022-03-17T19:34:45.2740540Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%] 2022-03-17T19:34:45.2773234Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%] 2022-03-17T19:34:45.2805422Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%] 2022-03-17T19:34:46.2850530Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%] 2022-03-17T19:34:47.2886502Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%] 2022-03-17T19:34:48.2925052Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED [ 83%] 2022-03-17T19:34:49.2957497Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%] 2022-03-17T19:34:49.2992798Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%] 2022-03-17T19:34:49.3023530Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED [100%] 2022-03-17T19:34:49.3024000Z 2022-03-17T19:34:49.3024600Z =============================== warnings summary =============================== 2022-03-17T19:34:49.3025233Z test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] 2022-03-17T19:34:49.3025793Z test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] 2022-03-17T19:34:49.3026829Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:34:49.3027675Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:34:49.3027894Z 2022-03-17T19:34:49.3028218Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2022-03-17T19:34:49.3029195Z =========================== short test summary info ============================ 2022-03-17T19:34:49.3029646Z SKIPPED [2] distributed/pipeline/sync/test_balance.py:47: Flaky due to time.sleep() 2022-03-17T19:34:49.3030917Z ================== 16 passed, 2 skipped, 2 warnings in 6.67s =================== 2022-03-17T19:34:49.5968965Z Running distributed/pipeline/sync/test_bugs ... [2022-03-17 19:34:49.596361] 2022-03-17T19:34:49.5969589Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-03-17 19:34:49.596473] 2022-03-17T19:34:50.5638956Z ============================= test session starts ============================== 2022-03-17T19:34:50.5639449Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:50.5660201Z cachedir: .pytest_cache 2022-03-17T19:34:50.5661384Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:50.5661760Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:50.5662081Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:50.5662498Z plugins: hypothesis-4.53.2 2022-03-17T19:34:50.5914566Z collecting ...  2022-03-17T19:34:50.5914894Z collected 4 items  2022-03-17T19:34:50.5919705Z 2022-03-17T19:34:50.6732312Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%] 2022-03-17T19:34:50.6914103Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED [ 50%] 2022-03-17T19:34:54.5048633Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait PASSED [ 75%] 2022-03-17T19:34:54.6442207Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED [100%] 2022-03-17T19:34:54.6443414Z 2022-03-17T19:34:54.6443861Z ============================== 4 passed in 4.08s =============================== 2022-03-17T19:34:54.8796099Z Running distributed/pipeline/sync/test_checkpoint ... [2022-03-17 19:34:54.879081] 2022-03-17T19:34:54.8796827Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-03-17 19:34:54.879187] 2022-03-17T19:34:55.8489274Z ============================= test session starts ============================== 2022-03-17T19:34:55.8489780Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:55.8514875Z cachedir: .pytest_cache 2022-03-17T19:34:55.8515902Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:55.8516261Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:55.8516592Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:55.8516992Z plugins: hypothesis-4.53.2 2022-03-17T19:34:55.8786428Z collecting ...  2022-03-17T19:34:55.8786910Z collected 9 items  2022-03-17T19:34:55.8791525Z 2022-03-17T19:34:55.8864266Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%] 2022-03-17T19:34:57.1341751Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%] 2022-03-17T19:34:57.1360885Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%] 2022-03-17T19:34:57.1379800Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%] 2022-03-17T19:34:57.1411696Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%] 2022-03-17T19:34:57.1448150Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%] 2022-03-17T19:34:57.1467869Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%] 2022-03-17T19:34:57.1483358Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%] 2022-03-17T19:34:57.1508942Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%] 2022-03-17T19:34:57.1510639Z 2022-03-17T19:34:57.1511230Z ============================== 9 passed in 1.30s =============================== 2022-03-17T19:34:57.3272660Z Running distributed/pipeline/sync/test_copy ... [2022-03-17 19:34:57.326776] 2022-03-17T19:34:57.3273203Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-03-17 19:34:57.326882] 2022-03-17T19:34:58.3014580Z ============================= test session starts ============================== 2022-03-17T19:34:58.3015103Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:34:58.3037077Z cachedir: .pytest_cache 2022-03-17T19:34:58.3037725Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:34:58.3038124Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:34:58.3038426Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:34:58.3038806Z plugins: hypothesis-4.53.2 2022-03-17T19:34:58.3280108Z collecting ...  2022-03-17T19:34:58.3280455Z collected 5 items  2022-03-17T19:34:58.3285180Z 2022-03-17T19:34:58.3344629Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED [ 20%] 2022-03-17T19:34:59.5990670Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED [ 40%] 2022-03-17T19:35:00.0585805Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED [ 60%] 2022-03-17T19:35:00.4350668Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED [ 80%] 2022-03-17T19:35:00.4373275Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%] 2022-03-17T19:35:00.4375203Z 2022-03-17T19:35:00.4375863Z ============================== 5 passed in 2.14s =============================== 2022-03-17T19:35:00.6167566Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-03-17 19:35:00.616231] 2022-03-17T19:35:00.6168234Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-03-17 19:35:00.616341] 2022-03-17T19:35:01.5920972Z ============================= test session starts ============================== 2022-03-17T19:35:01.5921532Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:01.5943188Z cachedir: .pytest_cache 2022-03-17T19:35:01.5943881Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:01.5944281Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:01.5944559Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:01.5944943Z plugins: hypothesis-4.53.2 2022-03-17T19:35:01.6248493Z collecting ...  2022-03-17T19:35:01.6248854Z collected 11 items  2022-03-17T19:35:01.6253082Z 2022-03-17T19:35:01.7055032Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [ 9%] 2022-03-17T19:35:01.7661775Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%] 2022-03-17T19:35:01.8160866Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%] 2022-03-17T19:35:01.8661928Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%] 2022-03-17T19:35:01.9002733Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%] 2022-03-17T19:35:01.9335169Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%] 2022-03-17T19:35:01.9358072Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%] 2022-03-17T19:35:01.9781307Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED [ 72%] 2022-03-17T19:35:02.1450902Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%] 2022-03-17T19:35:02.2552694Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%] 2022-03-17T19:35:02.2798661Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%] 2022-03-17T19:35:02.2799725Z 2022-03-17T19:35:02.2800051Z ============================== 11 passed in 0.69s ============================== 2022-03-17T19:35:02.3995777Z Running distributed/pipeline/sync/test_dependency ... [2022-03-17 19:35:02.399107] 2022-03-17T19:35:02.3996446Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-03-17 19:35:02.399216] 2022-03-17T19:35:03.3829480Z ============================= test session starts ============================== 2022-03-17T19:35:03.3830001Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:03.3851659Z cachedir: .pytest_cache 2022-03-17T19:35:03.3852567Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:03.3852933Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:03.3853266Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:03.3853844Z plugins: hypothesis-4.53.2 2022-03-17T19:35:03.4188358Z collecting ...  2022-03-17T19:35:03.4188738Z collected 6 items  2022-03-17T19:35:03.4193576Z 2022-03-17T19:35:04.6697810Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED [ 16%] 2022-03-17T19:35:04.6713048Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%] 2022-03-17T19:35:04.6731171Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%] 2022-03-17T19:35:04.6748389Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED [ 66%] 2022-03-17T19:35:04.6763093Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%] 2022-03-17T19:35:04.6783306Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%] 2022-03-17T19:35:04.6785152Z 2022-03-17T19:35:04.6785464Z ============================== 6 passed in 1.30s =============================== 2022-03-17T19:35:04.8532243Z Running distributed/pipeline/sync/test_inplace ... [2022-03-17 19:35:04.852730] 2022-03-17T19:35:04.8532862Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-03-17 19:35:04.852837] 2022-03-17T19:35:05.8236283Z ============================= test session starts ============================== 2022-03-17T19:35:05.8236809Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:05.8258218Z cachedir: .pytest_cache 2022-03-17T19:35:05.8258818Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:05.8259297Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:05.8259557Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:05.8259960Z plugins: hypothesis-4.53.2 2022-03-17T19:35:05.8389374Z collecting ...  2022-03-17T19:35:05.8389833Z collected 3 items  2022-03-17T19:35:05.8394401Z 2022-03-17T19:35:05.9270343Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%] 2022-03-17T19:35:05.9428691Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%] 2022-03-17T19:35:05.9592614Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%] 2022-03-17T19:35:05.9597059Z 2022-03-17T19:35:05.9597492Z =========================== short test summary info ============================ 2022-03-17T19:35:05.9598362Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad 2022-03-17T19:35:05.9598936Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad 2022-03-17T19:35:05.9600385Z ========================= 1 passed, 2 xfailed in 0.14s ========================= 2022-03-17T19:35:06.0702981Z Running distributed/pipeline/sync/test_microbatch ... [2022-03-17 19:35:06.069786] 2022-03-17T19:35:06.0703825Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-03-17 19:35:06.069896] 2022-03-17T19:35:07.0209561Z ============================= test session starts ============================== 2022-03-17T19:35:07.0210112Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:07.0231967Z cachedir: .pytest_cache 2022-03-17T19:35:07.0232553Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:07.0232975Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:07.0233323Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:07.0233963Z plugins: hypothesis-4.53.2 2022-03-17T19:35:07.0551692Z collecting ...  2022-03-17T19:35:07.0552145Z collected 10 items  2022-03-17T19:35:07.0556466Z 2022-03-17T19:35:07.0590618Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED [ 10%] 2022-03-17T19:35:07.0607021Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%] 2022-03-17T19:35:07.0623833Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED [ 30%] 2022-03-17T19:35:07.0640953Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%] 2022-03-17T19:35:07.0657663Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%] 2022-03-17T19:35:07.0678030Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED [ 60%] 2022-03-17T19:35:07.0704903Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%] 2022-03-17T19:35:07.0721782Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED [ 80%] 2022-03-17T19:35:07.0738718Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%] 2022-03-17T19:35:07.0760417Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%] 2022-03-17T19:35:07.0762102Z 2022-03-17T19:35:07.0762663Z ============================== 10 passed in 0.06s ============================== 2022-03-17T19:35:07.1784675Z Running distributed/pipeline/sync/test_phony ... [2022-03-17 19:35:07.177939] 2022-03-17T19:35:07.1785310Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-03-17 19:35:07.178077] 2022-03-17T19:35:08.1361445Z ============================= test session starts ============================== 2022-03-17T19:35:08.1362035Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:08.1383840Z cachedir: .pytest_cache 2022-03-17T19:35:08.1384394Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:08.1384846Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:08.1385141Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:08.1385578Z plugins: hypothesis-4.53.2 2022-03-17T19:35:08.1547552Z collecting ...  2022-03-17T19:35:08.1547948Z collected 4 items  2022-03-17T19:35:08.1552631Z 2022-03-17T19:35:08.1588774Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED [ 25%] 2022-03-17T19:35:08.1605853Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%] 2022-03-17T19:35:08.1622615Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED [ 75%] 2022-03-17T19:35:08.1656143Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%] 2022-03-17T19:35:08.1658049Z 2022-03-17T19:35:08.1658789Z ============================== 4 passed in 0.03s =============================== 2022-03-17T19:35:08.2728621Z Running distributed/pipeline/sync/test_pipe ... [2022-03-17 19:35:08.272378] 2022-03-17T19:35:08.2729261Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-03-17 19:35:08.272489] 2022-03-17T19:35:09.2213710Z ============================= test session starts ============================== 2022-03-17T19:35:09.2214261Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:09.2236126Z cachedir: .pytest_cache 2022-03-17T19:35:09.2236997Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:09.2237409Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:09.2237745Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:09.2238135Z plugins: hypothesis-4.53.2 2022-03-17T19:35:09.3474328Z collecting ...  2022-03-17T19:35:09.3474789Z collected 56 items  2022-03-17T19:35:09.3478437Z 2022-03-17T19:35:09.3524179Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED [ 1%] 2022-03-17T19:35:09.4288275Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED [ 3%] 2022-03-17T19:35:09.4445690Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED [ 5%] 2022-03-17T19:35:09.4605719Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED [ 7%] 2022-03-17T19:35:09.4757828Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED [ 8%] 2022-03-17T19:35:09.4940116Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%] 2022-03-17T19:35:09.5108949Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED [ 12%] 2022-03-17T19:35:09.5301027Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED [ 14%] 2022-03-17T19:35:09.5454518Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%] 2022-03-17T19:35:09.5616753Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%] 2022-03-17T19:35:09.5790730Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED [ 19%] 2022-03-17T19:35:09.5972211Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%] 2022-03-17T19:35:09.6136164Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED [ 23%] 2022-03-17T19:35:09.6293712Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED [ 25%] 2022-03-17T19:35:09.8502431Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%] 2022-03-17T19:35:09.8691008Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED [ 28%] 2022-03-17T19:35:09.8870619Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED [ 30%] 2022-03-17T19:35:09.9040654Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%] 2022-03-17T19:35:09.9213504Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED [ 33%] 2022-03-17T19:35:09.9372061Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED [ 35%] 2022-03-17T19:35:09.9528197Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED [ 37%] 2022-03-17T19:35:09.9693685Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED [ 39%] 2022-03-17T19:35:09.9932361Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%] 2022-03-17T19:35:10.0221764Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%] 2022-03-17T19:35:10.0501125Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%] 2022-03-17T19:35:10.0662267Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%] 2022-03-17T19:35:10.0823126Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%] 2022-03-17T19:35:10.0985615Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%] 2022-03-17T19:35:10.1158274Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%] 2022-03-17T19:35:10.1344099Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%] 2022-03-17T19:35:10.1524087Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%] 2022-03-17T19:35:10.1697520Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED [ 57%] 2022-03-17T19:35:10.1882540Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED [ 58%] 2022-03-17T19:35:10.2062842Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%] 2022-03-17T19:35:10.2326890Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%] 2022-03-17T19:35:10.2561492Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%] 2022-03-17T19:35:10.2783658Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%] 2022-03-17T19:35:10.2987573Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%] 2022-03-17T19:35:10.3203452Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%] 2022-03-17T19:35:10.3361753Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED [ 71%] 2022-03-17T19:35:10.3523509Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED [ 73%] 2022-03-17T19:35:11.6159838Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions PASSED [ 75%] 2022-03-17T19:35:11.6321496Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED [ 76%] 2022-03-17T19:35:11.6475060Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED [ 78%] 2022-03-17T19:35:11.6631112Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED [ 80%] 2022-03-17T19:35:11.6782113Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%] 2022-03-17T19:35:11.6937129Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%] 2022-03-17T19:35:11.7097107Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device PASSED [ 85%] 2022-03-17T19:35:13.2230577Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules PASSED [ 87%] 2022-03-17T19:35:13.2391609Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%] 2022-03-17T19:35:13.5582758Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED [ 91%] 2022-03-17T19:35:13.5758695Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] PASSED [ 92%] 2022-03-17T19:35:13.5940085Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] PASSED [ 94%] 2022-03-17T19:35:13.6119142Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] PASSED [ 96%] 2022-03-17T19:35:13.6282791Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device PASSED [ 98%] 2022-03-17T19:35:13.6908570Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper PASSED [100%] 2022-03-17T19:35:13.6908864Z 2022-03-17T19:35:13.6909141Z =============================== warnings summary =============================== 2022-03-17T19:35:13.6909577Z test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible 2022-03-17T19:35:13.6910171Z test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small 2022-03-17T19:35:13.6911229Z /opt/conda/lib/python3.7/site-packages/_pytest/python.py:192: PytestRemovedIn8Warning: Passing None has been deprecated. 2022-03-17T19:35:13.6912037Z See https://docs.pytest.org/en/latest/how-to/capture-warnings.html#additional-use-cases-of-warnings-in-tests for alternatives in common use cases. 2022-03-17T19:35:13.6912571Z result = testfunction(**testargs) 2022-03-17T19:35:13.6912816Z 2022-03-17T19:35:13.6913003Z test/distributed/pipeline/sync/test_pipe.py::test_partitions 2022-03-17T19:35:13.6913998Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:35:13.6914789Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:35:13.6915057Z 2022-03-17T19:35:13.6915339Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2022-03-17T19:35:13.6915900Z ======================== 56 passed, 3 warnings in 4.47s ======================== 2022-03-17T19:35:13.9522091Z Running distributed/pipeline/sync/test_pipeline ... [2022-03-17 19:35:13.951699] 2022-03-17T19:35:13.9522760Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-03-17 19:35:13.951809] 2022-03-17T19:35:14.9264780Z ============================= test session starts ============================== 2022-03-17T19:35:14.9265338Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:14.9286663Z cachedir: .pytest_cache 2022-03-17T19:35:14.9287273Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:14.9287697Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:14.9288051Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:14.9288415Z plugins: hypothesis-4.53.2 2022-03-17T19:35:14.9441498Z collecting ...  2022-03-17T19:35:14.9441904Z collected 1 item  2022-03-17T19:35:14.9446378Z 2022-03-17T19:35:14.9486477Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED [100%] 2022-03-17T19:35:14.9487386Z 2022-03-17T19:35:14.9487925Z ============================== 1 passed in 0.02s =============================== 2022-03-17T19:35:15.0572810Z Running distributed/pipeline/sync/test_stream ... [2022-03-17 19:35:15.056777] 2022-03-17T19:35:15.0573697Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-03-17 19:35:15.056887] 2022-03-17T19:35:16.0346875Z ============================= test session starts ============================== 2022-03-17T19:35:16.0347505Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:16.0368521Z cachedir: .pytest_cache 2022-03-17T19:35:16.0369103Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:16.0369856Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:16.0370171Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:16.0370563Z plugins: hypothesis-4.53.2 2022-03-17T19:35:16.0774425Z collecting ...  2022-03-17T19:35:16.0774849Z collected 19 items  2022-03-17T19:35:16.0779119Z 2022-03-17T19:35:16.0813023Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [ 5%] 2022-03-17T19:35:17.3280186Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%] 2022-03-17T19:35:17.3294567Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%] 2022-03-17T19:35:17.3309145Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%] 2022-03-17T19:35:17.3323198Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%] 2022-03-17T19:35:17.3337887Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%] 2022-03-17T19:35:17.3352242Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%] 2022-03-17T19:35:17.3366488Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%] 2022-03-17T19:35:17.3380177Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%] 2022-03-17T19:35:17.3395695Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%] 2022-03-17T19:35:17.3409612Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%] 2022-03-17T19:35:17.3424388Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%] 2022-03-17T19:35:17.3642891Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%] 2022-03-17T19:35:17.8061393Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%] 2022-03-17T19:35:17.8078081Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%] 2022-03-17T19:35:18.1533278Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%] 2022-03-17T19:35:18.1548720Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%] 2022-03-17T19:35:18.4443313Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%] 2022-03-17T19:35:18.4472745Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%] 2022-03-17T19:35:18.4474598Z 2022-03-17T19:35:18.4475029Z ============================== 19 passed in 2.41s ============================== 2022-03-17T19:35:18.7227843Z Running distributed/pipeline/sync/test_transparency ... [2022-03-17 19:35:18.722277] 2022-03-17T19:35:18.7228494Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-03-17 19:35:18.722382] 2022-03-17T19:35:19.6927157Z ============================= test session starts ============================== 2022-03-17T19:35:19.6927720Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:19.6948883Z cachedir: .pytest_cache 2022-03-17T19:35:19.6949506Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:19.6949932Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:19.6950558Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:19.6950951Z plugins: hypothesis-4.53.2 2022-03-17T19:35:19.7068303Z collecting ...  2022-03-17T19:35:19.7068757Z collected 1 item  2022-03-17T19:35:19.7073219Z 2022-03-17T19:35:19.8232532Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%] 2022-03-17T19:35:19.8233164Z 2022-03-17T19:35:19.8233524Z ============================== 1 passed in 0.13s =============================== 2022-03-17T19:35:19.9330780Z Running distributed/pipeline/sync/test_worker ... [2022-03-17 19:35:19.932555] 2022-03-17T19:35:19.9331445Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-03-17 19:35:19.932663] 2022-03-17T19:35:20.8844697Z ============================= test session starts ============================== 2022-03-17T19:35:20.8845217Z platform linux -- Python 3.7.11, pytest-7.1.0, pluggy-1.0.0 -- /opt/conda/bin/python 2022-03-17T19:35:20.8866854Z cachedir: .pytest_cache 2022-03-17T19:35:20.8867574Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples') 2022-03-17T19:35:20.8868032Z torch: 1.12.0a0+git4b4f652 2022-03-17T19:35:20.8868601Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini 2022-03-17T19:35:20.8868880Z plugins: hypothesis-4.53.2 2022-03-17T19:35:20.9080432Z collecting ...  2022-03-17T19:35:20.9080941Z collected 6 items  2022-03-17T19:35:20.9085359Z 2022-03-17T19:35:20.9128906Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%] 2022-03-17T19:35:20.9152047Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED [ 33%] 2022-03-17T19:35:20.9172508Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED [ 50%] 2022-03-17T19:35:20.9205064Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED [ 66%] 2022-03-17T19:35:20.9227575Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED [ 83%] 2022-03-17T19:35:20.9257389Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED [100%] 2022-03-17T19:35:20.9258961Z 2022-03-17T19:35:20.9259615Z ============================== 6 passed in 0.04s =============================== 2022-03-17T19:35:21.0369797Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-03-17 19:35:21.036450] 2022-03-17T19:35:21.0370636Z Executing ['/opt/conda/bin/python', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:35:21.036559] 2022-03-17T19:35:21.8191900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpva0yw6l_ 2022-03-17T19:35:21.8192947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpva0yw6l_/_remote_module_non_sriptable.py 2022-03-17T19:35:22.1636029Z ]> 2022-03-17T19:35:22.1636634Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) 2022-03-17T19:35:22.1637443Z , <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]> 2022-03-17T19:35:22.1638199Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) 2022-03-17T19:35:22.1638628Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) 2022-03-17T19:35:22.1639236Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) 2022-03-17T19:35:22.1640130Z , <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]> 2022-03-17T19:35:22.1640999Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) 2022-03-17T19:35:22.1641442Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) 2022-03-17T19:35:22.1642005Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) 2022-03-17T19:35:22.1642427Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) 2022-03-17T19:35:22.1642883Z ]> 2022-03-17T19:35:22.1643368Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) 2022-03-17T19:35:22.1644606Z , <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]> 2022-03-17T19:35:22.1645974Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2022-03-17T19:35:22.1646670Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2022-03-17T19:35:22.1647454Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2022-03-17T19:35:22.1648216Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2022-03-17T19:35:22.1648768Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) 2022-03-17T19:35:22.1649183Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) 2022-03-17T19:35:22.1649576Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) 2022-03-17T19:35:22.1649988Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) 2022-03-17T19:35:22.1665337Z , <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]> 2022-03-17T19:35:22.1679521Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1680052Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1680570Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1681141Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1681665Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1682206Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1682756Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1683308Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1683856Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1684353Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1684834Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1685369Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1685845Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1686346Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1686823Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1687322Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1687778Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1688290Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1688769Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1689227Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1689692Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1690156Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1690660Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1691149Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1691630Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1692124Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1692577Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1693045Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1693518Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1693966Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1694439Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1694883Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1695343Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1695821Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1696284Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1696756Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1697224Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1697698Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1698169Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1698624Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1699095Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1699554Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1700057Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1701069Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1701555Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1702034Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1702505Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1702980Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1703496Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1704090Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1704575Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1705058Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1705534Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1706041Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1706532Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1707120Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1707652Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1708155Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1708656Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1709112Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1709588Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1710053Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1710504Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1710991Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1711470Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1711946Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1712417Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1712943Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1713457Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1713957Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1714471Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1714983Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1715467Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1715964Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1716433Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1716918Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1717408Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1717884Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1718383Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1718882Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1719361Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1719858Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1720336Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1720823Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1721313Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1721863Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1722358Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) 2022-03-17T19:35:22.1723234Z , <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]> 2022-03-17T19:35:22.1724135Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-03-17T19:35:22.1724703Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-03-17T19:35:22.1725214Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) 2022-03-17T19:35:22.9519376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdnraj68 2022-03-17T19:35:22.9520587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdnraj68/_remote_module_non_sriptable.py 2022-03-17T19:35:23.3080260Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:35:23.3095263Z 2022-03-17T19:35:23.3095515Z Running tests... 2022-03-17T19:35:23.3095961Z ---------------------------------------------------------------------- 2022-03-17T19:35:24.8428381Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:35:24.8840553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45270 2022-03-17T19:35:24.8959139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45271 2022-03-17T19:35:24.9079995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45272 2022-03-17T19:35:24.9207203Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45273 2022-03-17T19:35:25.7037380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_m3j5fa 2022-03-17T19:35:25.7038025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_m3j5fa/_remote_module_non_sriptable.py 2022-03-17T19:35:25.7052173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ykzokpy 2022-03-17T19:35:25.7055057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ykzokpy/_remote_module_non_sriptable.py 2022-03-17T19:35:25.7353081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfga0nu7d 2022-03-17T19:35:25.7355461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfga0nu7d/_remote_module_non_sriptable.py 2022-03-17T19:35:25.7588989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgg4a33b4 2022-03-17T19:35:25.7591321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgg4a33b4/_remote_module_non_sriptable.py 2022-03-17T19:35:26.0569894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:26.0612223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:26.0783744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:35:26.1111109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:35:26.3252814Z skip (3.015s) 2022-03-17T19:35:26.3253071Z 2022-03-17T19:35:26.3253450Z ---------------------------------------------------------------------- 2022-03-17T19:35:26.3253785Z Ran 1 test in 3.016s 2022-03-17T19:35:26.3253929Z 2022-03-17T19:35:26.3254016Z OK (skipped=1) 2022-03-17T19:35:26.3254164Z 2022-03-17T19:35:26.3254292Z Generating XML reports... 2022-03-17T19:35:26.3297289Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220317193523.xml 2022-03-17T19:35:27.3692762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1duk8v2 2022-03-17T19:35:27.3693983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1duk8v2/_remote_module_non_sriptable.py 2022-03-17T19:35:27.7208696Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:35:27.7224901Z 2022-03-17T19:35:27.7225039Z Running tests... 2022-03-17T19:35:27.7225772Z ---------------------------------------------------------------------- 2022-03-17T19:35:29.2523000Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:35:29.2937471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45451 2022-03-17T19:35:29.3055185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45452 2022-03-17T19:35:29.3177303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45453 2022-03-17T19:35:29.3301243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45454 2022-03-17T19:35:30.1165834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qj6rr_9 2022-03-17T19:35:30.1172443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qj6rr_9/_remote_module_non_sriptable.py 2022-03-17T19:35:30.1698613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tu_z58s 2022-03-17T19:35:30.1699914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tu_z58s/_remote_module_non_sriptable.py 2022-03-17T19:35:30.1702514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjugjj13q 2022-03-17T19:35:30.1706664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjugjj13q/_remote_module_non_sriptable.py 2022-03-17T19:35:30.1900113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgpctyc2 2022-03-17T19:35:30.1902773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgpctyc2/_remote_module_non_sriptable.py 2022-03-17T19:35:30.4643859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:30.5139027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:35:30.5165781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:30.5392874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:35:32.8392604Z ok (5.116s) 2022-03-17T19:35:32.8393016Z 2022-03-17T19:35:32.8393408Z ---------------------------------------------------------------------- 2022-03-17T19:35:32.8393735Z Ran 1 test in 5.117s 2022-03-17T19:35:32.8393888Z 2022-03-17T19:35:32.8393982Z OK 2022-03-17T19:35:32.8394100Z 2022-03-17T19:35:32.8394240Z Generating XML reports... 2022-03-17T19:35:32.8438657Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220317193527.xml 2022-03-17T19:35:33.8938300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvfi77mb 2022-03-17T19:35:33.8939237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvfi77mb/_remote_module_non_sriptable.py 2022-03-17T19:35:34.2516836Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:35:34.2532324Z 2022-03-17T19:35:34.2532631Z Running tests... 2022-03-17T19:35:34.2533061Z ---------------------------------------------------------------------- 2022-03-17T19:35:35.7780721Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:35:35.8195381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45880 2022-03-17T19:35:35.8314749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45881 2022-03-17T19:35:35.8434915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45882 2022-03-17T19:35:35.8558337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45883 2022-03-17T19:35:36.6387319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdptvrigv 2022-03-17T19:35:36.6388281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdptvrigv/_remote_module_non_sriptable.py 2022-03-17T19:35:36.6941879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhnpuk5i 2022-03-17T19:35:36.6944474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhnpuk5i/_remote_module_non_sriptable.py 2022-03-17T19:35:36.6979754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmyorwd8 2022-03-17T19:35:36.6982257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmyorwd8/_remote_module_non_sriptable.py 2022-03-17T19:35:36.7190406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnte_fz0 2022-03-17T19:35:36.7193148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnte_fz0/_remote_module_non_sriptable.py 2022-03-17T19:35:36.9945883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:37.0393301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:37.0402382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:35:37.0660045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:35:39.3649110Z ok (5.111s) 2022-03-17T19:35:39.3649272Z 2022-03-17T19:35:39.3649770Z ---------------------------------------------------------------------- 2022-03-17T19:35:39.3650124Z Ran 1 test in 5.112s 2022-03-17T19:35:39.3650272Z 2022-03-17T19:35:39.3650358Z OK 2022-03-17T19:35:39.3650492Z 2022-03-17T19:35:39.3650558Z Generating XML reports... 2022-03-17T19:35:39.3693217Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220317193534.xml 2022-03-17T19:35:40.4144913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp969kspk0 2022-03-17T19:35:40.4145958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp969kspk0/_remote_module_non_sriptable.py 2022-03-17T19:35:40.7687888Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:35:40.7703319Z 2022-03-17T19:35:40.7703574Z Running tests... 2022-03-17T19:35:40.7704017Z ---------------------------------------------------------------------- 2022-03-17T19:35:42.3018323Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:35:42.3446762Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46309 2022-03-17T19:35:42.3564700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46310 2022-03-17T19:35:42.3686728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46311 2022-03-17T19:35:42.3813967Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46312 2022-03-17T19:35:43.1491803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezbmccb0 2022-03-17T19:35:43.1492919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezbmccb0/_remote_module_non_sriptable.py 2022-03-17T19:35:43.1891506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6ijak73 2022-03-17T19:35:43.1892977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6ijak73/_remote_module_non_sriptable.py 2022-03-17T19:35:43.2431560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8lrmveg 2022-03-17T19:35:43.2433241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8lrmveg/_remote_module_non_sriptable.py 2022-03-17T19:35:43.2490410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nnix702 2022-03-17T19:35:43.2492615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nnix702/_remote_module_non_sriptable.py 2022-03-17T19:35:43.4988865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:43.5366237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:43.5911437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:35:43.5912142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:35:45.8904227Z ok (5.120s) 2022-03-17T19:35:45.8904609Z 2022-03-17T19:35:45.8905280Z ---------------------------------------------------------------------- 2022-03-17T19:35:45.8905866Z Ran 1 test in 5.120s 2022-03-17T19:35:45.8906167Z 2022-03-17T19:35:45.8906355Z OK 2022-03-17T19:35:45.8906588Z 2022-03-17T19:35:45.8906804Z Generating XML reports... 2022-03-17T19:35:45.8951087Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220317193540.xml 2022-03-17T19:35:46.9385191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14_xq7l1 2022-03-17T19:35:46.9386293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14_xq7l1/_remote_module_non_sriptable.py 2022-03-17T19:35:47.2930290Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:35:47.2946158Z 2022-03-17T19:35:47.2946289Z Running tests... 2022-03-17T19:35:47.2946912Z ---------------------------------------------------------------------- 2022-03-17T19:35:48.8289312Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:35:48.8709016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46738 2022-03-17T19:35:48.8829177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46739 2022-03-17T19:35:49.7364441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps1ygjbq_ 2022-03-17T19:35:49.7365268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps1ygjbq_/_remote_module_non_sriptable.py 2022-03-17T19:35:49.7366108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wh7o636 2022-03-17T19:35:49.7369115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wh7o636/_remote_module_non_sriptable.py 2022-03-17T19:35:50.0846546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:50.0874348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:51.7906594Z ok (4.496s) 2022-03-17T19:35:51.7906992Z 2022-03-17T19:35:51.7907401Z ---------------------------------------------------------------------- 2022-03-17T19:35:51.7907766Z Ran 1 test in 4.496s 2022-03-17T19:35:51.7907932Z 2022-03-17T19:35:51.7908022Z OK 2022-03-17T19:35:51.7908137Z 2022-03-17T19:35:51.7908275Z Generating XML reports... 2022-03-17T19:35:51.7951239Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220317193547.xml 2022-03-17T19:35:52.8447740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpir6s8qpk 2022-03-17T19:35:52.8448913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpir6s8qpk/_remote_module_non_sriptable.py 2022-03-17T19:35:53.1991225Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:35:53.2006270Z 2022-03-17T19:35:53.2006565Z Running tests... 2022-03-17T19:35:53.2007024Z ---------------------------------------------------------------------- 2022-03-17T19:35:54.7414775Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:35:54.7832502Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46932 2022-03-17T19:35:54.7952677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46933 2022-03-17T19:35:55.6514390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34irmhn_ 2022-03-17T19:35:55.6515510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34irmhn_/_remote_module_non_sriptable.py 2022-03-17T19:35:55.6626728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi95lwrti 2022-03-17T19:35:55.6629019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi95lwrti/_remote_module_non_sriptable.py 2022-03-17T19:35:55.9963606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:35:56.0114098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:35:56.1823066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34irmhn_/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-03-17T19:35:56.1823856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi95lwrti/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-03-17T19:35:56.1931145Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpi95lwrti/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py 2022-03-17T19:35:57.9026565Z ok (4.702s) 2022-03-17T19:35:57.9026881Z 2022-03-17T19:35:57.9027256Z ---------------------------------------------------------------------- 2022-03-17T19:35:57.9027586Z Ran 1 test in 4.702s 2022-03-17T19:35:57.9027754Z 2022-03-17T19:35:57.9027815Z OK 2022-03-17T19:35:57.9027968Z 2022-03-17T19:35:57.9028999Z Generating XML reports... 2022-03-17T19:35:57.9071218Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220317193553.xml 2022-03-17T19:35:58.9480266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusyb09pa 2022-03-17T19:35:58.9481739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusyb09pa/_remote_module_non_sriptable.py 2022-03-17T19:35:59.3060854Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:35:59.3076307Z 2022-03-17T19:35:59.3076424Z Running tests... 2022-03-17T19:35:59.3077254Z ---------------------------------------------------------------------- 2022-03-17T19:36:00.8484204Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:00.8906257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47142 2022-03-17T19:36:00.9028797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47143 2022-03-17T19:36:01.7019859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9215n8m 2022-03-17T19:36:01.7020916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9215n8m/_remote_module_non_sriptable.py 2022-03-17T19:36:01.7150214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_l3hs2r 2022-03-17T19:36:01.7153141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_l3hs2r/_remote_module_non_sriptable.py 2022-03-17T19:36:02.0514093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:02.0624050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:02.2487396Z On WorkerInfo(id=1, name=worker1): 2022-03-17T19:36:02.2505615Z RuntimeError('CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7ff7debc50bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #1: + 0x147c4 (0x7ff7e78d67c4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)\nframe #2: + 0xf04786 (0x7ff7dfd04786 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #3: + 0x2981484 (0x7ff7e1781484 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #4: + 0x298156b (0x7ff7e178156b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x14f (0x7ff7e9185acf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: + 0x18c3c5f (0x7ff7e93d8c5f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #7: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x1ff (0x7ff7e91c810f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: at::native::_to_copy(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x13f6 (0x7ff7e8c8d0d6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: + 0x1a12af3 (0x7ff7e9527af3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: + 0x18c56f7 (0x7ff7e93da6f7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: + 0x263a875 (0x7ff7ea14f875 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: + 0x263ae0b (0x7ff7ea14fe0b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x1c9 (0x7ff7e8fce819 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: at::native::to(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x13e (0x7ff7e8c8b94e in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: + 0x1ac9ed9 (0x7ff7e95deed9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x1e0 (0x7ff7e90db6f0 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: + 0x2d092a (0x7ff7f47bf92a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #20: + 0x2d0e12 (0x7ff7f47bfe12 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #21: _PyMethodDescr_FastCallKeywords + 0x330 (0x55cea932c6c0 in /opt/conda/bin/python)\nframe #22: + 0x1801ae (0x55cea932d1ae in /opt/conda/bin/python)\nframe #23: _PyEval_EvalFrameDefault + 0x621 (0x55cea9370381 in /opt/conda/bin/python)\nframe #24: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python)\nframe #25: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python)\nframe #26: _PyEval_EvalFrameDefault + 0x3f5 (0x55cea9370155 in /opt/conda/bin/python)\nframe #27: _PyFunction_FastCallKeywords + 0x187 (0x55cea92e5d17 in /opt/conda/bin/python)\nframe #28: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python)\nframe #29: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python)\nframe #30: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python)\nframe #31: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python)\nframe #32: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python)\nframe #33: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python)\nframe #34: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python)\nframe #35: _PyEval_EvalFrameDefault + 0x1d0d (0x55cea9371a6d in /opt/conda/bin/python)\nframe #36: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python)\nframe #37: + 0x8f2a34 (0x7ff7f4de1a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #38: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7ff7f4de03dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #39: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7ff7f4de2db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x83 (0x7ff7f4de3403 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7ff7eb1d75d4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7ff7f4de2ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #43: + 0x36bc0f3 (0x7ff7eb1d10f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7ff7eb1d1cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #45: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7ff7eb1cc597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: + 0x36ea602 (0x7ff7eb1ff602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7ff7debb15eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #48: + 0xc9039 (0x7ff801d3a039 in /opt/conda/lib/libstdc++.so.6)\nframe #49: + 0x76ba (0x7ff8227ef6ba in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #50: clone + 0x6d (0x7ff82252551d in /lib/x86_64-linux-gnu/libc.so.6)\n') 2022-03-17T19:36:02.2516059Z Traceback (most recent call last): 2022-03-17T19:36:02.2516722Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-03-17T19:36:02.2517178Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-03-17T19:36:02.2517813Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 87, in _create_module 2022-03-17T19:36:02.2518190Z module.to(device) 2022-03-17T19:36:02.2518641Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 907, in to 2022-03-17T19:36:02.2518986Z return self._apply(convert) 2022-03-17T19:36:02.2519464Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 601, in _apply 2022-03-17T19:36:02.2519832Z param_applied = fn(param) 2022-03-17T19:36:02.2520281Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 905, in convert 2022-03-17T19:36:02.2520738Z return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) 2022-03-17T19:36:02.2521116Z RuntimeError: CUDA error: invalid device ordinal 2022-03-17T19:36:02.2521561Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. 2022-03-17T19:36:02.2522012Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 2022-03-17T19:36:02.2522462Z Exception raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first): 2022-03-17T19:36:02.2523300Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7ff7debc50bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:36:02.2524031Z frame #1: + 0x147c4 (0x7ff7e78d67c4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) 2022-03-17T19:36:02.2524641Z frame #2: + 0xf04786 (0x7ff7dfd04786 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2525274Z frame #3: + 0x2981484 (0x7ff7e1781484 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2525880Z frame #4: + 0x298156b (0x7ff7e178156b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2526866Z frame #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x14f (0x7ff7e9185acf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2527705Z frame #6: + 0x18c3c5f (0x7ff7e93d8c5f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2528630Z frame #7: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x1ff (0x7ff7e91c810f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2529703Z frame #8: at::native::_to_copy(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x13f6 (0x7ff7e8c8d0d6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2530500Z frame #9: + 0x1a12af3 (0x7ff7e9527af3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2531577Z frame #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2532396Z frame #11: + 0x18c56f7 (0x7ff7e93da6f7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2533439Z frame #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2534281Z frame #13: + 0x263a875 (0x7ff7ea14f875 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2534916Z frame #14: + 0x263ae0b (0x7ff7ea14fe0b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2535843Z frame #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x1c9 (0x7ff7e8fce819 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2536943Z frame #16: at::native::to(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x13e (0x7ff7e8c8b94e in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2537751Z frame #17: + 0x1ac9ed9 (0x7ff7e95deed9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2538697Z frame #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x1e0 (0x7ff7e90db6f0 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2539530Z frame #19: + 0x2d092a (0x7ff7f47bf92a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2540170Z frame #20: + 0x2d0e12 (0x7ff7f47bfe12 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2540973Z frame #21: _PyMethodDescr_FastCallKeywords + 0x330 (0x55cea932c6c0 in /opt/conda/bin/python) 2022-03-17T19:36:02.2541407Z frame #22: + 0x1801ae (0x55cea932d1ae in /opt/conda/bin/python) 2022-03-17T19:36:02.2541818Z frame #23: _PyEval_EvalFrameDefault + 0x621 (0x55cea9370381 in /opt/conda/bin/python) 2022-03-17T19:36:02.2542222Z frame #24: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python) 2022-03-17T19:36:02.2542655Z frame #25: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python) 2022-03-17T19:36:02.2543060Z frame #26: _PyEval_EvalFrameDefault + 0x3f5 (0x55cea9370155 in /opt/conda/bin/python) 2022-03-17T19:36:02.2543480Z frame #27: _PyFunction_FastCallKeywords + 0x187 (0x55cea92e5d17 in /opt/conda/bin/python) 2022-03-17T19:36:02.2543890Z frame #28: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python) 2022-03-17T19:36:02.2544278Z frame #29: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python) 2022-03-17T19:36:02.2544689Z frame #30: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python) 2022-03-17T19:36:02.2545100Z frame #31: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python) 2022-03-17T19:36:02.2545509Z frame #32: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python) 2022-03-17T19:36:02.2545912Z frame #33: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python) 2022-03-17T19:36:02.2546416Z frame #34: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python) 2022-03-17T19:36:02.2546835Z frame #35: _PyEval_EvalFrameDefault + 0x1d0d (0x55cea9371a6d in /opt/conda/bin/python) 2022-03-17T19:36:02.2547232Z frame #36: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python) 2022-03-17T19:36:02.2547847Z frame #37: + 0x8f2a34 (0x7ff7f4de1a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2548699Z frame #38: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7ff7f4de03dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2549692Z frame #39: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7ff7f4de2db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2550825Z frame #40: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x83 (0x7ff7f4de3403 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2552031Z frame #41: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7ff7eb1d75d4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2553294Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7ff7f4de2ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2554158Z frame #43: + 0x36bc0f3 (0x7ff7eb1d10f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2555080Z frame #44: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7ff7eb1d1cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2556139Z frame #45: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7ff7eb1cc597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2556913Z frame #46: + 0x36ea602 (0x7ff7eb1ff602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2557566Z frame #47: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7ff7debb15eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:36:02.2558059Z frame #48: + 0xc9039 (0x7ff801d3a039 in /opt/conda/lib/libstdc++.so.6) 2022-03-17T19:36:02.2558576Z frame #49: + 0x76ba (0x7ff8227ef6ba in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-03-17T19:36:02.2559068Z frame #50: clone + 0x6d (0x7ff82252551d in /lib/x86_64-linux-gnu/libc.so.6) 2022-03-17T19:36:02.2559286Z 2022-03-17T19:36:02.2559305Z 2022-03-17T19:36:02.2559438Z On WorkerInfo(id=1, name=worker1): 2022-03-17T19:36:02.2595334Z RuntimeError('On WorkerInfo(id=1, name=worker1):\nRuntimeError(\'CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7ff7debc50bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #1: + 0x147c4 (0x7ff7e78d67c4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)\nframe #2: + 0xf04786 (0x7ff7dfd04786 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #3: + 0x2981484 (0x7ff7e1781484 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #4: + 0x298156b (0x7ff7e178156b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x14f (0x7ff7e9185acf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: + 0x18c3c5f (0x7ff7e93d8c5f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #7: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x1ff (0x7ff7e91c810f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: at::native::_to_copy(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x13f6 (0x7ff7e8c8d0d6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: + 0x1a12af3 (0x7ff7e9527af3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: + 0x18c56f7 (0x7ff7e93da6f7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: + 0x263a875 (0x7ff7ea14f875 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: + 0x263ae0b (0x7ff7ea14fe0b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x1c9 (0x7ff7e8fce819 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: at::native::to(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x13e (0x7ff7e8c8b94e in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: + 0x1ac9ed9 (0x7ff7e95deed9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x1e0 (0x7ff7e90db6f0 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: + 0x2d092a (0x7ff7f47bf92a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #20: + 0x2d0e12 (0x7ff7f47bfe12 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #21: _PyMethodDescr_FastCallKeywords + 0x330 (0x55cea932c6c0 in /opt/conda/bin/python)\nframe #22: + 0x1801ae (0x55cea932d1ae in /opt/conda/bin/python)\nframe #23: _PyEval_EvalFrameDefault + 0x621 (0x55cea9370381 in /opt/conda/bin/python)\nframe #24: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python)\nframe #25: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python)\nframe #26: _PyEval_EvalFrameDefault + 0x3f5 (0x55cea9370155 in /opt/conda/bin/python)\nframe #27: _PyFunction_FastCallKeywords + 0x187 (0x55cea92e5d17 in /opt/conda/bin/python)\nframe #28: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python)\nframe #29: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python)\nframe #30: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python)\nframe #31: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python)\nframe #32: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python)\nframe #33: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python)\nframe #34: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python)\nframe #35: _PyEval_EvalFrameDefault + 0x1d0d (0x55cea9371a6d in /opt/conda/bin/python)\nframe #36: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python)\nframe #37: + 0x8f2a34 (0x7ff7f4de1a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #38: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7ff7f4de03dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #39: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7ff7f4de2db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x83 (0x7ff7f4de3403 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7ff7eb1d75d4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7ff7f4de2ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #43: + 0x36bc0f3 (0x7ff7eb1d10f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7ff7eb1d1cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #45: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7ff7eb1cc597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: + 0x36ea602 (0x7ff7eb1ff602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7ff7debb15eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #48: + 0xc9039 (0x7ff801d3a039 in /opt/conda/lib/libstdc++.so.6)\nframe #49: + 0x76ba (0x7ff8227ef6ba in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #50: clone + 0x6d (0x7ff82252551d in /lib/x86_64-linux-gnu/libc.so.6)\n\')\nTraceback (most recent call last):\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 87, in _create_module\n module.to(device)\n File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 907, in to\n return self._apply(convert)\n File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 601, in _apply\n param_applied = fn(param)\n File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 905, in convert\n return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)\nRuntimeError: CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7ff7debc50bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #1: + 0x147c4 (0x7ff7e78d67c4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)\nframe #2: + 0xf04786 (0x7ff7dfd04786 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #3: + 0x2981484 (0x7ff7e1781484 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #4: + 0x298156b (0x7ff7e178156b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x14f (0x7ff7e9185acf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: + 0x18c3c5f (0x7ff7e93d8c5f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #7: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x1ff (0x7ff7e91c810f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: at::native::_to_copy(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x13f6 (0x7ff7e8c8d0d6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: + 0x1a12af3 (0x7ff7e9527af3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: + 0x18c56f7 (0x7ff7e93da6f7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: + 0x263a875 (0x7ff7ea14f875 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: + 0x263ae0b (0x7ff7ea14fe0b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x1c9 (0x7ff7e8fce819 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: at::native::to(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x13e (0x7ff7e8c8b94e in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: + 0x1ac9ed9 (0x7ff7e95deed9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x1e0 (0x7ff7e90db6f0 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: + 0x2d092a (0x7ff7f47bf92a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #20: + 0x2d0e12 (0x7ff7f47bfe12 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #21: _PyMethodDescr_FastCallKeywords + 0x330 (0x55cea932c6c0 in /opt/conda/bin/python)\nframe #22: + 0x1801ae (0x55cea932d1ae in /opt/conda/bin/python)\nframe #23: _PyEval_EvalFrameDefault + 0x621 (0x55cea9370381 in /opt/conda/bin/python)\nframe #24: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python)\nframe #25: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python)\nframe #26: _PyEval_EvalFrameDefault + 0x3f5 (0x55cea9370155 in /opt/conda/bin/python)\nframe #27: _PyFunction_FastCallKeywords + 0x187 (0x55cea92e5d17 in /opt/conda/bin/python)\nframe #28: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python)\nframe #29: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python)\nframe #30: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python)\nframe #31: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python)\nframe #32: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python)\nframe #33: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python)\nframe #34: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python)\nframe #35: _PyEval_EvalFrameDefault + 0x1d0d (0x55cea9371a6d in /opt/conda/bin/python)\nframe #36: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python)\nframe #37: + 0x8f2a34 (0x7ff7f4de1a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #38: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7ff7f4de03dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #39: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7ff7f4de2db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x83 (0x7ff7f4de3403 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7ff7eb1d75d4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7ff7f4de2ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #43: + 0x36bc0f3 (0x7ff7eb1d10f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7ff7eb1d1cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #45: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7ff7eb1cc597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: + 0x36ea602 (0x7ff7eb1ff602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7ff7debb15eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #48: + 0xc9039 (0x7ff801d3a039 in /opt/conda/lib/libstdc++.so.6)\nframe #49: + 0x76ba (0x7ff8227ef6ba in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #50: clone + 0x6d (0x7ff82252551d in /lib/x86_64-linux-gnu/libc.so.6)\n\n') 2022-03-17T19:36:02.2617272Z Traceback (most recent call last): 2022-03-17T19:36:02.2617806Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-03-17T19:36:02.2618240Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-03-17T19:36:02.2618657Z File "/tmp/tmpusyb09pa/_remote_module_non_sriptable.py", line 47, in _remote_forward 2022-03-17T19:36:02.2619030Z module = module_rref.local_value() 2022-03-17T19:36:02.2619536Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception 2022-03-17T19:36:02.2620108Z raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape")) 2022-03-17T19:36:02.2620675Z RuntimeError: On WorkerInfo(id=1, name=worker1): 2022-03-17T19:36:02.2621082Z RuntimeError('CUDA error: invalid device ordinal 2022-03-17T19:36:02.2621509Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. 2022-03-17T19:36:02.2621961Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 2022-03-17T19:36:02.2622433Z Exception raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first): 2022-03-17T19:36:02.2623257Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7ff7debc50bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:36:02.2623986Z frame #1: + 0x147c4 (0x7ff7e78d67c4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) 2022-03-17T19:36:02.2624618Z frame #2: + 0xf04786 (0x7ff7dfd04786 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2625242Z frame #3: + 0x2981484 (0x7ff7e1781484 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2625873Z frame #4: + 0x298156b (0x7ff7e178156b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2626859Z frame #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x14f (0x7ff7e9185acf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2627678Z frame #6: + 0x18c3c5f (0x7ff7e93d8c5f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2628602Z frame #7: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x1ff (0x7ff7e91c810f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2629794Z frame #8: at::native::_to_copy(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x13f6 (0x7ff7e8c8d0d6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2630586Z frame #9: + 0x1a12af3 (0x7ff7e9527af3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2631627Z frame #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2632483Z frame #11: + 0x18c56f7 (0x7ff7e93da6f7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2633480Z frame #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2634296Z frame #13: + 0x263a875 (0x7ff7ea14f875 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2634935Z frame #14: + 0x263ae0b (0x7ff7ea14fe0b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2635875Z frame #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x1c9 (0x7ff7e8fce819 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2636965Z frame #16: at::native::to(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x13e (0x7ff7e8c8b94e in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2637766Z frame #17: + 0x1ac9ed9 (0x7ff7e95deed9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2638738Z frame #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x1e0 (0x7ff7e90db6f0 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2639566Z frame #19: + 0x2d092a (0x7ff7f47bf92a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2640192Z frame #20: + 0x2d0e12 (0x7ff7f47bfe12 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2640673Z frame #21: _PyMethodDescr_FastCallKeywords + 0x330 (0x55cea932c6c0 in /opt/conda/bin/python) 2022-03-17T19:36:02.2641081Z frame #22: + 0x1801ae (0x55cea932d1ae in /opt/conda/bin/python) 2022-03-17T19:36:02.2641491Z frame #23: _PyEval_EvalFrameDefault + 0x621 (0x55cea9370381 in /opt/conda/bin/python) 2022-03-17T19:36:02.2641915Z frame #24: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python) 2022-03-17T19:36:02.2642325Z frame #25: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python) 2022-03-17T19:36:02.2642746Z frame #26: _PyEval_EvalFrameDefault + 0x3f5 (0x55cea9370155 in /opt/conda/bin/python) 2022-03-17T19:36:02.2643163Z frame #27: _PyFunction_FastCallKeywords + 0x187 (0x55cea92e5d17 in /opt/conda/bin/python) 2022-03-17T19:36:02.2643626Z frame #28: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python) 2022-03-17T19:36:02.2644032Z frame #29: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python) 2022-03-17T19:36:02.2644435Z frame #30: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python) 2022-03-17T19:36:02.2644853Z frame #31: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python) 2022-03-17T19:36:02.2645265Z frame #32: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python) 2022-03-17T19:36:02.2645654Z frame #33: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python) 2022-03-17T19:36:02.2646123Z frame #34: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python) 2022-03-17T19:36:02.2646534Z frame #35: _PyEval_EvalFrameDefault + 0x1d0d (0x55cea9371a6d in /opt/conda/bin/python) 2022-03-17T19:36:02.2646952Z frame #36: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python) 2022-03-17T19:36:02.2647565Z frame #37: + 0x8f2a34 (0x7ff7f4de1a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2648327Z frame #38: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7ff7f4de03dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2649325Z frame #39: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7ff7f4de2db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2650450Z frame #40: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x83 (0x7ff7f4de3403 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2651633Z frame #41: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7ff7eb1d75d4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2652896Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7ff7f4de2ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2653786Z frame #43: + 0x36bc0f3 (0x7ff7eb1d10f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2654718Z frame #44: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7ff7eb1d1cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2655755Z frame #45: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7ff7eb1cc597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2656532Z frame #46: + 0x36ea602 (0x7ff7eb1ff602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2657210Z frame #47: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7ff7debb15eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:36:02.2657688Z frame #48: + 0xc9039 (0x7ff801d3a039 in /opt/conda/lib/libstdc++.so.6) 2022-03-17T19:36:02.2658225Z frame #49: + 0x76ba (0x7ff8227ef6ba in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-03-17T19:36:02.2658718Z frame #50: clone + 0x6d (0x7ff82252551d in /lib/x86_64-linux-gnu/libc.so.6) 2022-03-17T19:36:02.2659092Z ') 2022-03-17T19:36:02.2659340Z Traceback (most recent call last): 2022-03-17T19:36:02.2659835Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-03-17T19:36:02.2660479Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-03-17T19:36:02.2661061Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 87, in _create_module 2022-03-17T19:36:02.2661422Z module.to(device) 2022-03-17T19:36:02.2661869Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 907, in to 2022-03-17T19:36:02.2662212Z return self._apply(convert) 2022-03-17T19:36:02.2662760Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 601, in _apply 2022-03-17T19:36:02.2663124Z param_applied = fn(param) 2022-03-17T19:36:02.2663596Z File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 905, in convert 2022-03-17T19:36:02.2664059Z return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) 2022-03-17T19:36:02.2664437Z RuntimeError: CUDA error: invalid device ordinal 2022-03-17T19:36:02.2664881Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. 2022-03-17T19:36:02.2665331Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1. 2022-03-17T19:36:02.2665782Z Exception raised from exchangeDevice at /var/lib/jenkins/workspace/c10/cuda/impl/CUDAGuardImpl.h:33 (most recent call first): 2022-03-17T19:36:02.2666620Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7ff7debc50bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:36:02.2667331Z frame #1: + 0x147c4 (0x7ff7e78d67c4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) 2022-03-17T19:36:02.2667970Z frame #2: + 0xf04786 (0x7ff7dfd04786 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2668607Z frame #3: + 0x2981484 (0x7ff7e1781484 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2669210Z frame #4: + 0x298156b (0x7ff7e178156b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:36:02.2670197Z frame #5: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x14f (0x7ff7e9185acf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2671028Z frame #6: + 0x18c3c5f (0x7ff7e93d8c5f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2671937Z frame #7: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optional, c10::optional, c10::optional, c10::optional) + 0x1ff (0x7ff7e91c810f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2673035Z frame #8: at::native::_to_copy(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x13f6 (0x7ff7e8c8d0d6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2673835Z frame #9: + 0x1a12af3 (0x7ff7e9527af3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2674835Z frame #10: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2675741Z frame #11: + 0x18c56f7 (0x7ff7e93da6f7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2676740Z frame #12: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x152 (0x7ff7e8f585f2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2677573Z frame #13: + 0x263a875 (0x7ff7ea14f875 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2678234Z frame #14: + 0x263ae0b (0x7ff7ea14fe0b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2679190Z frame #15: at::_ops::_to_copy::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, c10::optional) + 0x1c9 (0x7ff7e8fce819 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2680296Z frame #16: at::native::to(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x13e (0x7ff7e8c8b94e in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2681096Z frame #17: + 0x1ac9ed9 (0x7ff7e95deed9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2682050Z frame #18: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional, c10::optional, c10::optional, c10::optional, bool, bool, c10::optional) + 0x1e0 (0x7ff7e90db6f0 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2682879Z frame #19: + 0x2d092a (0x7ff7f47bf92a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2683524Z frame #20: + 0x2d0e12 (0x7ff7f47bfe12 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2683990Z frame #21: _PyMethodDescr_FastCallKeywords + 0x330 (0x55cea932c6c0 in /opt/conda/bin/python) 2022-03-17T19:36:02.2684417Z frame #22: + 0x1801ae (0x55cea932d1ae in /opt/conda/bin/python) 2022-03-17T19:36:02.2684808Z frame #23: _PyEval_EvalFrameDefault + 0x621 (0x55cea9370381 in /opt/conda/bin/python) 2022-03-17T19:36:02.2685233Z frame #24: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python) 2022-03-17T19:36:02.2685662Z frame #25: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python) 2022-03-17T19:36:02.2686066Z frame #26: _PyEval_EvalFrameDefault + 0x3f5 (0x55cea9370155 in /opt/conda/bin/python) 2022-03-17T19:36:02.2686484Z frame #27: _PyFunction_FastCallKeywords + 0x187 (0x55cea92e5d17 in /opt/conda/bin/python) 2022-03-17T19:36:02.2686900Z frame #28: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python) 2022-03-17T19:36:02.2687290Z frame #29: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python) 2022-03-17T19:36:02.2687708Z frame #30: _PyEval_EvalCodeWithName + 0xc5c (0x55cea92c659c in /opt/conda/bin/python) 2022-03-17T19:36:02.2688118Z frame #31: _PyFunction_FastCallKeywords + 0x693 (0x55cea92e6223 in /opt/conda/bin/python) 2022-03-17T19:36:02.2688529Z frame #32: + 0x1800c5 (0x55cea932d0c5 in /opt/conda/bin/python) 2022-03-17T19:36:02.2688933Z frame #33: _PyEval_EvalFrameDefault + 0x48a2 (0x55cea9374602 in /opt/conda/bin/python) 2022-03-17T19:36:02.2689328Z frame #34: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python) 2022-03-17T19:36:02.2689744Z frame #35: _PyEval_EvalFrameDefault + 0x1d0d (0x55cea9371a6d in /opt/conda/bin/python) 2022-03-17T19:36:02.2690205Z frame #36: _PyFunction_FastCallDict + 0x118 (0x55cea92e5138 in /opt/conda/bin/python) 2022-03-17T19:36:02.2690806Z frame #37: + 0x8f2a34 (0x7ff7f4de1a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2691586Z frame #38: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7ff7f4de03dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2692615Z frame #39: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7ff7f4de2db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2693751Z frame #40: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x83 (0x7ff7f4de3403 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2694959Z frame #41: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x194 (0x7ff7eb1d75d4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2696223Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7ff7f4de2ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:36:02.2697086Z frame #43: + 0x36bc0f3 (0x7ff7eb1d10f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2698016Z frame #44: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7ff7eb1d1cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2699081Z frame #45: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7ff7eb1cc597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2699846Z frame #46: + 0x36ea602 (0x7ff7eb1ff602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:36:02.2700745Z frame #47: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7ff7debb15eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:36:02.2701242Z frame #48: + 0xc9039 (0x7ff801d3a039 in /opt/conda/lib/libstdc++.so.6) 2022-03-17T19:36:02.2701764Z frame #49: + 0x76ba (0x7ff8227ef6ba in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-03-17T19:36:02.2702264Z frame #50: clone + 0x6d (0x7ff82252551d in /lib/x86_64-linux-gnu/libc.so.6) 2022-03-17T19:36:02.2702488Z 2022-03-17T19:36:02.2702506Z 2022-03-17T19:36:02.2702526Z 2022-03-17T19:36:02.4072564Z ok (3.099s) 2022-03-17T19:36:02.4072814Z 2022-03-17T19:36:02.4073171Z ---------------------------------------------------------------------- 2022-03-17T19:36:02.4073511Z Ran 1 test in 3.100s 2022-03-17T19:36:02.4073683Z 2022-03-17T19:36:02.4073753Z OK 2022-03-17T19:36:02.4073870Z 2022-03-17T19:36:02.4074012Z Generating XML reports... 2022-03-17T19:36:02.4115886Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220317193559.xml 2022-03-17T19:36:03.4571504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0cbbzcl 2022-03-17T19:36:03.4572379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0cbbzcl/_remote_module_non_sriptable.py 2022-03-17T19:36:03.8110736Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:03.8126026Z 2022-03-17T19:36:03.8126233Z Running tests... 2022-03-17T19:36:03.8126835Z ---------------------------------------------------------------------- 2022-03-17T19:36:05.3546129Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:05.3962365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47335 2022-03-17T19:36:05.4080631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47336 2022-03-17T19:36:06.1966714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpox6cdqvr 2022-03-17T19:36:06.1967870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpox6cdqvr/_remote_module_non_sriptable.py 2022-03-17T19:36:06.2015269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdv01nswy 2022-03-17T19:36:06.2018430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdv01nswy/_remote_module_non_sriptable.py 2022-03-17T19:36:06.5467438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:06.5499619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:08.2152781Z ok (4.402s) 2022-03-17T19:36:08.2153037Z 2022-03-17T19:36:08.2153391Z ---------------------------------------------------------------------- 2022-03-17T19:36:08.2153738Z Ran 1 test in 4.403s 2022-03-17T19:36:08.2153897Z 2022-03-17T19:36:08.2153965Z OK 2022-03-17T19:36:08.2154097Z 2022-03-17T19:36:08.2154218Z Generating XML reports... 2022-03-17T19:36:08.2199468Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220317193603.xml 2022-03-17T19:36:09.2564574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxht_o1r 2022-03-17T19:36:09.2565410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxht_o1r/_remote_module_non_sriptable.py 2022-03-17T19:36:09.6047288Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:09.6063304Z 2022-03-17T19:36:09.6063481Z Running tests... 2022-03-17T19:36:09.6064146Z ---------------------------------------------------------------------- 2022-03-17T19:36:11.1277058Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:11.1686817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47529 2022-03-17T19:36:11.1806342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47530 2022-03-17T19:36:11.1927246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47531 2022-03-17T19:36:11.2048626Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47532 2022-03-17T19:36:11.9831491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5v4bwgx 2022-03-17T19:36:11.9833008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5v4bwgx/_remote_module_non_sriptable.py 2022-03-17T19:36:12.0434600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq7fissb 2022-03-17T19:36:12.0436979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq7fissb/_remote_module_non_sriptable.py 2022-03-17T19:36:12.0485989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_ksm6r8 2022-03-17T19:36:12.0489407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_ksm6r8/_remote_module_non_sriptable.py 2022-03-17T19:36:12.0525166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6xweuef 2022-03-17T19:36:12.0527864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6xweuef/_remote_module_non_sriptable.py 2022-03-17T19:36:12.3320343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:12.3950410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:36:12.3951558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:36:12.3958533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:16.7171728Z ok (7.110s) 2022-03-17T19:36:16.7171917Z 2022-03-17T19:36:16.7172300Z ---------------------------------------------------------------------- 2022-03-17T19:36:16.7172660Z Ran 1 test in 7.111s 2022-03-17T19:36:16.7172792Z 2022-03-17T19:36:16.7172862Z OK 2022-03-17T19:36:16.7172980Z 2022-03-17T19:36:16.7173414Z Generating XML reports... 2022-03-17T19:36:16.7216256Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220317193609.xml 2022-03-17T19:36:17.7593826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a7qoosh 2022-03-17T19:36:17.7595386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a7qoosh/_remote_module_non_sriptable.py 2022-03-17T19:36:18.1160105Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:18.1175468Z 2022-03-17T19:36:18.1175856Z Running tests... 2022-03-17T19:36:18.1176325Z ---------------------------------------------------------------------- 2022-03-17T19:36:19.6509429Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:19.6926987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47882 2022-03-17T19:36:19.7045947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47883 2022-03-17T19:36:20.5151757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nckgv8n 2022-03-17T19:36:20.5152371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nckgv8n/_remote_module_non_sriptable.py 2022-03-17T19:36:20.5420187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvdzy_m0 2022-03-17T19:36:20.5422801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvdzy_m0/_remote_module_non_sriptable.py 2022-03-17T19:36:20.8719304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:20.8841012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:21.0085326Z skip (2.891s) 2022-03-17T19:36:21.0085577Z 2022-03-17T19:36:21.0086200Z ---------------------------------------------------------------------- 2022-03-17T19:36:21.0086520Z Ran 1 test in 2.891s 2022-03-17T19:36:21.0086656Z 2022-03-17T19:36:21.0086771Z OK (skipped=1) 2022-03-17T19:36:21.0086908Z 2022-03-17T19:36:21.0087036Z Generating XML reports... 2022-03-17T19:36:21.0130054Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193618.xml 2022-03-17T19:36:22.0337342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2lhebbk 2022-03-17T19:36:22.0338461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2lhebbk/_remote_module_non_sriptable.py 2022-03-17T19:36:22.3932900Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:22.3948432Z 2022-03-17T19:36:22.3948931Z Running tests... 2022-03-17T19:36:22.3949351Z ---------------------------------------------------------------------- 2022-03-17T19:36:23.9291632Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:23.9707714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47991 2022-03-17T19:36:23.9829712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47992 2022-03-17T19:36:24.8041441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpini3x7tb 2022-03-17T19:36:24.8042288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpini3x7tb/_remote_module_non_sriptable.py 2022-03-17T19:36:24.8254793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7o4h0er2 2022-03-17T19:36:24.8257311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7o4h0er2/_remote_module_non_sriptable.py 2022-03-17T19:36:25.1488031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:25.1731168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:25.2870642Z skip (2.892s) 2022-03-17T19:36:25.2871079Z 2022-03-17T19:36:25.2871492Z ---------------------------------------------------------------------- 2022-03-17T19:36:25.2871847Z Ran 1 test in 2.892s 2022-03-17T19:36:25.2872010Z 2022-03-17T19:36:25.2872102Z OK (skipped=1) 2022-03-17T19:36:25.2872268Z 2022-03-17T19:36:25.2872394Z Generating XML reports... 2022-03-17T19:36:25.2914855Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193622.xml 2022-03-17T19:36:26.3215899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpto3at1tx 2022-03-17T19:36:26.3217095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpto3at1tx/_remote_module_non_sriptable.py 2022-03-17T19:36:26.6751576Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:26.6766659Z 2022-03-17T19:36:26.6766904Z Running tests... 2022-03-17T19:36:26.6767338Z ---------------------------------------------------------------------- 2022-03-17T19:36:28.2131720Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:28.2547760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48100 2022-03-17T19:36:28.2667072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48101 2022-03-17T19:36:29.0797858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksel_gbz 2022-03-17T19:36:29.0798951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksel_gbz/_remote_module_non_sriptable.py 2022-03-17T19:36:29.1116810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpziz0_3he 2022-03-17T19:36:29.1119696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpziz0_3he/_remote_module_non_sriptable.py 2022-03-17T19:36:29.4244390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:29.4681740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:29.6708620Z skip (2.994s) 2022-03-17T19:36:29.6708805Z 2022-03-17T19:36:29.6709174Z ---------------------------------------------------------------------- 2022-03-17T19:36:29.6709542Z Ran 1 test in 2.994s 2022-03-17T19:36:29.6709715Z 2022-03-17T19:36:29.6709826Z OK (skipped=1) 2022-03-17T19:36:29.6709982Z 2022-03-17T19:36:29.6710089Z Generating XML reports... 2022-03-17T19:36:29.6753789Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193626.xml 2022-03-17T19:36:30.7248307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpveujx1b2 2022-03-17T19:36:30.7249450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpveujx1b2/_remote_module_non_sriptable.py 2022-03-17T19:36:31.0844165Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:31.0859499Z 2022-03-17T19:36:31.0859704Z Running tests... 2022-03-17T19:36:31.0860143Z ---------------------------------------------------------------------- 2022-03-17T19:36:32.6121511Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:32.6535491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48209 2022-03-17T19:36:32.6656520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48210 2022-03-17T19:36:33.4580505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2jpyn35 2022-03-17T19:36:33.4581328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2dswp3aa 2022-03-17T19:36:33.4581876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2jpyn35/_remote_module_non_sriptable.py 2022-03-17T19:36:33.4582589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2dswp3aa/_remote_module_non_sriptable.py 2022-03-17T19:36:33.8060022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:33.8071607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:33.9701306Z skip (2.884s) 2022-03-17T19:36:33.9701532Z 2022-03-17T19:36:33.9701916Z ---------------------------------------------------------------------- 2022-03-17T19:36:33.9702230Z Ran 1 test in 2.884s 2022-03-17T19:36:33.9702398Z 2022-03-17T19:36:33.9704199Z OK (skipped=1) 2022-03-17T19:36:33.9704530Z 2022-03-17T19:36:33.9704835Z Generating XML reports... 2022-03-17T19:36:33.9745615Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193631.xml 2022-03-17T19:36:35.0181936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe99so3gl 2022-03-17T19:36:35.0184480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe99so3gl/_remote_module_non_sriptable.py 2022-03-17T19:36:35.3720935Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:35.3735973Z 2022-03-17T19:36:35.3736217Z Running tests... 2022-03-17T19:36:35.3736652Z ---------------------------------------------------------------------- 2022-03-17T19:36:36.9260644Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:36.9678462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48318 2022-03-17T19:36:36.9797564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48319 2022-03-17T19:36:37.7625858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf40ajd91 2022-03-17T19:36:37.7626793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf40ajd91/_remote_module_non_sriptable.py 2022-03-17T19:36:37.7660921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp48bkf_2 2022-03-17T19:36:37.7664207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp48bkf_2/_remote_module_non_sriptable.py 2022-03-17T19:36:38.1124587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:38.1127672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:38.2837965Z skip (2.910s) 2022-03-17T19:36:38.2838216Z 2022-03-17T19:36:38.2838633Z ---------------------------------------------------------------------- 2022-03-17T19:36:38.2838966Z Ran 1 test in 2.910s 2022-03-17T19:36:38.2839115Z 2022-03-17T19:36:38.2839226Z OK (skipped=1) 2022-03-17T19:36:38.2839370Z 2022-03-17T19:36:38.2839505Z Generating XML reports... 2022-03-17T19:36:38.2883243Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193635.xml 2022-03-17T19:36:39.3310883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nb71_gd 2022-03-17T19:36:39.3311949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nb71_gd/_remote_module_non_sriptable.py 2022-03-17T19:36:39.6854711Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:39.6870213Z 2022-03-17T19:36:39.6870565Z Running tests... 2022-03-17T19:36:39.6871275Z ---------------------------------------------------------------------- 2022-03-17T19:36:41.2263186Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:41.2680039Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48427 2022-03-17T19:36:41.2800001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48428 2022-03-17T19:36:42.0644007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzsnu3m2 2022-03-17T19:36:42.0644887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzsnu3m2/_remote_module_non_sriptable.py 2022-03-17T19:36:42.0784716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4bd7w5s 2022-03-17T19:36:42.0787655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4bd7w5s/_remote_module_non_sriptable.py 2022-03-17T19:36:42.4222932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:42.4314135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:42.5841032Z skip (2.897s) 2022-03-17T19:36:42.5841278Z 2022-03-17T19:36:42.5841675Z ---------------------------------------------------------------------- 2022-03-17T19:36:42.5842018Z Ran 1 test in 2.897s 2022-03-17T19:36:42.5842176Z 2022-03-17T19:36:42.5842289Z OK (skipped=1) 2022-03-17T19:36:42.5842446Z 2022-03-17T19:36:42.5842595Z Generating XML reports... 2022-03-17T19:36:42.5886002Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193639.xml 2022-03-17T19:36:43.6272686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rcgabia 2022-03-17T19:36:43.6273536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rcgabia/_remote_module_non_sriptable.py 2022-03-17T19:36:43.9796990Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:43.9811814Z 2022-03-17T19:36:43.9812226Z Running tests... 2022-03-17T19:36:43.9812724Z ---------------------------------------------------------------------- 2022-03-17T19:36:45.5052672Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:45.5471124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48536 2022-03-17T19:36:45.5590034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48537 2022-03-17T19:36:46.3779847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhahqw56 2022-03-17T19:36:46.3780995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhahqw56/_remote_module_non_sriptable.py 2022-03-17T19:36:46.4059147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn90z2ry 2022-03-17T19:36:46.4062274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn90z2ry/_remote_module_non_sriptable.py 2022-03-17T19:36:46.7283871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:46.7589191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:46.9635928Z skip (2.982s) 2022-03-17T19:36:46.9636292Z 2022-03-17T19:36:46.9636770Z ---------------------------------------------------------------------- 2022-03-17T19:36:46.9637072Z Ran 1 test in 2.982s 2022-03-17T19:36:46.9637244Z 2022-03-17T19:36:46.9637356Z OK (skipped=1) 2022-03-17T19:36:46.9637512Z 2022-03-17T19:36:46.9637639Z Generating XML reports... 2022-03-17T19:36:46.9682329Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193643.xml 2022-03-17T19:36:48.0083795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxr9jv4vq 2022-03-17T19:36:48.0084662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxr9jv4vq/_remote_module_non_sriptable.py 2022-03-17T19:36:48.3591171Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:48.3606349Z 2022-03-17T19:36:48.3606671Z Running tests... 2022-03-17T19:36:48.3607024Z ---------------------------------------------------------------------- 2022-03-17T19:36:49.8675880Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:49.9089355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48645 2022-03-17T19:36:49.9206813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48646 2022-03-17T19:36:50.7101952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3h9io19 2022-03-17T19:36:50.7103027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3h9io19/_remote_module_non_sriptable.py 2022-03-17T19:36:50.7586629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzj9rfgzj 2022-03-17T19:36:50.7588892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzj9rfgzj/_remote_module_non_sriptable.py 2022-03-17T19:36:51.0599935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:51.1039020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:51.2246920Z skip (2.864s) 2022-03-17T19:36:51.2247158Z 2022-03-17T19:36:51.2247546Z ---------------------------------------------------------------------- 2022-03-17T19:36:51.2247858Z Ran 1 test in 2.864s 2022-03-17T19:36:51.2248017Z 2022-03-17T19:36:51.2248139Z OK (skipped=1) 2022-03-17T19:36:51.2248317Z 2022-03-17T19:36:51.2248438Z Generating XML reports... 2022-03-17T19:36:51.2291768Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193648.xml 2022-03-17T19:36:52.2515450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydh_mcyi 2022-03-17T19:36:52.2516781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydh_mcyi/_remote_module_non_sriptable.py 2022-03-17T19:36:52.6091629Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:36:52.6107355Z 2022-03-17T19:36:52.6107603Z Running tests... 2022-03-17T19:36:52.6108053Z ---------------------------------------------------------------------- 2022-03-17T19:36:54.1378366Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:36:54.1803337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48754 2022-03-17T19:36:54.1921507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48755 2022-03-17T19:36:54.2042194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48756 2022-03-17T19:36:54.2167730Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48757 2022-03-17T19:36:55.0263260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm3ekfnf 2022-03-17T19:36:55.0264450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm3ekfnf/_remote_module_non_sriptable.py 2022-03-17T19:36:55.0656389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46y5p6xs 2022-03-17T19:36:55.0659171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46y5p6xs/_remote_module_non_sriptable.py 2022-03-17T19:36:55.0875764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4px3x7d 2022-03-17T19:36:55.0878855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4px3x7d/_remote_module_non_sriptable.py 2022-03-17T19:36:55.0935822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspg8i1bt 2022-03-17T19:36:55.0938737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspg8i1bt/_remote_module_non_sriptable.py 2022-03-17T19:36:55.3693929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:36:55.4070655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:36:55.4296727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:36:55.4394640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:37:00.2297134Z ok (7.619s) 2022-03-17T19:37:00.2297403Z 2022-03-17T19:37:00.2297914Z ---------------------------------------------------------------------- 2022-03-17T19:37:00.2298231Z Ran 1 test in 7.619s 2022-03-17T19:37:00.2298502Z 2022-03-17T19:37:00.2298541Z OK 2022-03-17T19:37:00.2298616Z 2022-03-17T19:37:00.2298748Z Generating XML reports... 2022-03-17T19:37:00.2341571Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193652.xml 2022-03-17T19:37:01.2798415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xpalvmj 2022-03-17T19:37:01.2799391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xpalvmj/_remote_module_non_sriptable.py 2022-03-17T19:37:01.6333835Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:37:01.6349376Z 2022-03-17T19:37:01.6349511Z Running tests... 2022-03-17T19:37:01.6350151Z ---------------------------------------------------------------------- 2022-03-17T19:37:03.1762645Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:37:03.2180051Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49107 2022-03-17T19:37:03.2301817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49108 2022-03-17T19:37:03.2427878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49109 2022-03-17T19:37:03.2551203Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49110 2022-03-17T19:37:04.0306494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsrgrjhd 2022-03-17T19:37:04.0307304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsrgrjhd/_remote_module_non_sriptable.py 2022-03-17T19:37:04.0397593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb67yx1dl 2022-03-17T19:37:04.0400405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb67yx1dl/_remote_module_non_sriptable.py 2022-03-17T19:37:04.0668431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5iqixnsp 2022-03-17T19:37:04.0671278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5iqixnsp/_remote_module_non_sriptable.py 2022-03-17T19:37:04.0769920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsahtlihe 2022-03-17T19:37:04.0772797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsahtlihe/_remote_module_non_sriptable.py 2022-03-17T19:37:04.3840311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:37:04.3840747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:04.4112960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:04.4251667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:37:11.7721540Z ok (10.137s) 2022-03-17T19:37:11.7721916Z 2022-03-17T19:37:11.7723004Z ---------------------------------------------------------------------- 2022-03-17T19:37:11.7723621Z Ran 1 test in 10.137s 2022-03-17T19:37:11.7723911Z 2022-03-17T19:37:11.7724086Z OK 2022-03-17T19:37:11.7726266Z 2022-03-17T19:37:11.7726521Z Generating XML reports... 2022-03-17T19:37:11.7768140Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193701.xml 2022-03-17T19:37:12.8180852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal0dxsf2 2022-03-17T19:37:12.8181529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal0dxsf2/_remote_module_non_sriptable.py 2022-03-17T19:37:13.1798139Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:37:13.1813099Z 2022-03-17T19:37:13.1813560Z Running tests... 2022-03-17T19:37:13.1814298Z ---------------------------------------------------------------------- 2022-03-17T19:37:14.7167619Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:37:14.7584006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49460 2022-03-17T19:37:14.7703007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49461 2022-03-17T19:37:14.7824926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49462 2022-03-17T19:37:14.7952554Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49463 2022-03-17T19:37:15.5996033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1csompoi 2022-03-17T19:37:15.5997178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1csompoi/_remote_module_non_sriptable.py 2022-03-17T19:37:15.6042519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpty77l_7w 2022-03-17T19:37:15.6044882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpty77l_7w/_remote_module_non_sriptable.py 2022-03-17T19:37:15.6262805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyu6kmp7c 2022-03-17T19:37:15.6265629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyu6kmp7c/_remote_module_non_sriptable.py 2022-03-17T19:37:15.6322449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzj_ribnc 2022-03-17T19:37:15.6325264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzj_ribnc/_remote_module_non_sriptable.py 2022-03-17T19:37:15.9481078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:15.9490615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:37:15.9825024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:37:15.9848756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:22.9116277Z ok (9.730s) 2022-03-17T19:37:22.9116508Z 2022-03-17T19:37:22.9117152Z ---------------------------------------------------------------------- 2022-03-17T19:37:22.9117482Z Ran 1 test in 9.730s 2022-03-17T19:37:22.9117636Z 2022-03-17T19:37:22.9117706Z OK 2022-03-17T19:37:22.9117842Z 2022-03-17T19:37:22.9117956Z Generating XML reports... 2022-03-17T19:37:22.9162050Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193713.xml 2022-03-17T19:37:23.9550552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperws9wv2 2022-03-17T19:37:23.9551649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperws9wv2/_remote_module_non_sriptable.py 2022-03-17T19:37:24.3097428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:37:24.3113255Z 2022-03-17T19:37:24.3113662Z Running tests... 2022-03-17T19:37:24.3114076Z ---------------------------------------------------------------------- 2022-03-17T19:37:25.8354700Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:37:25.8769509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49649 2022-03-17T19:37:25.8890066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49650 2022-03-17T19:37:25.9012533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49651 2022-03-17T19:37:25.9136857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49652 2022-03-17T19:37:26.7255893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22sjma3p 2022-03-17T19:37:26.7256998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22sjma3p/_remote_module_non_sriptable.py 2022-03-17T19:37:26.7803990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi54omgs6 2022-03-17T19:37:26.7806046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi54omgs6/_remote_module_non_sriptable.py 2022-03-17T19:37:26.8379089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6ygunx3 2022-03-17T19:37:26.8381599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6ygunx3/_remote_module_non_sriptable.py 2022-03-17T19:37:26.8605297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28nuvuvg 2022-03-17T19:37:26.8608546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28nuvuvg/_remote_module_non_sriptable.py 2022-03-17T19:37:27.0672080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:27.1282134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:27.1896282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:37:27.2127081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:37:32.7279135Z ok (8.416s) 2022-03-17T19:37:32.7282122Z 2022-03-17T19:37:32.7283032Z ---------------------------------------------------------------------- 2022-03-17T19:37:32.7283666Z Ran 1 test in 8.416s 2022-03-17T19:37:32.7283965Z 2022-03-17T19:37:32.7284150Z OK 2022-03-17T19:37:32.7284394Z 2022-03-17T19:37:32.7284620Z Generating XML reports... 2022-03-17T19:37:32.7330856Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193724.xml 2022-03-17T19:37:33.7727151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptafiiori 2022-03-17T19:37:33.7728265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptafiiori/_remote_module_non_sriptable.py 2022-03-17T19:37:34.1250507Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:37:34.1266054Z 2022-03-17T19:37:34.1266208Z Running tests... 2022-03-17T19:37:34.1266939Z ---------------------------------------------------------------------- 2022-03-17T19:37:35.6585298Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:37:35.7005836Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49894 2022-03-17T19:37:35.7125493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49895 2022-03-17T19:37:35.7248359Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49896 2022-03-17T19:37:35.7377780Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49897 2022-03-17T19:37:36.5345965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rticfx2 2022-03-17T19:37:36.5346923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rticfx2/_remote_module_non_sriptable.py 2022-03-17T19:37:36.5521321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxt93g0mq 2022-03-17T19:37:36.5523826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxt93g0mq/_remote_module_non_sriptable.py 2022-03-17T19:37:36.5756966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5609go2 2022-03-17T19:37:36.5760013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5609go2/_remote_module_non_sriptable.py 2022-03-17T19:37:36.6189956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphh0r33ld 2022-03-17T19:37:36.6192880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphh0r33ld/_remote_module_non_sriptable.py 2022-03-17T19:37:36.8819063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:36.8942143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:36.9244263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:37:36.9722338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:37:41.0494067Z ok (6.922s) 2022-03-17T19:37:41.0494377Z 2022-03-17T19:37:41.0494731Z ---------------------------------------------------------------------- 2022-03-17T19:37:41.0495050Z Ran 1 test in 6.923s 2022-03-17T19:37:41.0495202Z 2022-03-17T19:37:41.0496655Z OK 2022-03-17T19:37:41.0497262Z 2022-03-17T19:37:41.0497623Z Generating XML reports... 2022-03-17T19:37:41.0540144Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193734.xml 2022-03-17T19:37:42.0882188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuob_rfq 2022-03-17T19:37:42.0883355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuob_rfq/_remote_module_non_sriptable.py 2022-03-17T19:37:42.4335691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:37:42.4350786Z 2022-03-17T19:37:42.4351186Z Running tests... 2022-03-17T19:37:42.4351868Z ---------------------------------------------------------------------- 2022-03-17T19:37:43.9268712Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:37:43.9678774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50079 2022-03-17T19:37:43.9797988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50080 2022-03-17T19:37:43.9920605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50081 2022-03-17T19:37:44.0042325Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50082 2022-03-17T19:37:44.8299057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprezu7sr7 2022-03-17T19:37:44.8300020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprezu7sr7/_remote_module_non_sriptable.py 2022-03-17T19:37:44.8635279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43r88_tj 2022-03-17T19:37:44.8635871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa223xv9g 2022-03-17T19:37:44.8638002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43r88_tj/_remote_module_non_sriptable.py 2022-03-17T19:37:44.8638534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa223xv9g/_remote_module_non_sriptable.py 2022-03-17T19:37:44.8667202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphizrv_wq 2022-03-17T19:37:44.8669792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphizrv_wq/_remote_module_non_sriptable.py 2022-03-17T19:37:45.1816899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:45.2081496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:37:45.2164625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:45.2167625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:37:50.9190160Z ok (8.484s) 2022-03-17T19:37:50.9190370Z 2022-03-17T19:37:50.9190740Z ---------------------------------------------------------------------- 2022-03-17T19:37:50.9191063Z Ran 1 test in 8.484s 2022-03-17T19:37:50.9191224Z 2022-03-17T19:37:50.9191308Z OK 2022-03-17T19:37:50.9191443Z 2022-03-17T19:37:50.9191897Z Generating XML reports... 2022-03-17T19:37:50.9235016Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193742.xml 2022-03-17T19:37:51.9612802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl83wmqyj 2022-03-17T19:37:51.9613633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl83wmqyj/_remote_module_non_sriptable.py 2022-03-17T19:37:52.3207209Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:37:52.3222626Z 2022-03-17T19:37:52.3222916Z Running tests... 2022-03-17T19:37:52.3223414Z ---------------------------------------------------------------------- 2022-03-17T19:37:53.8329954Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:37:53.8744156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50324 2022-03-17T19:37:53.8863199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50325 2022-03-17T19:37:53.8984886Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50326 2022-03-17T19:37:53.9108810Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50327 2022-03-17T19:37:54.6905345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6h183bxe 2022-03-17T19:37:54.6906281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6h183bxe/_remote_module_non_sriptable.py 2022-03-17T19:37:54.7426384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7yhyyr42 2022-03-17T19:37:54.7428587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7yhyyr42/_remote_module_non_sriptable.py 2022-03-17T19:37:54.7742368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknd4c3f5 2022-03-17T19:37:54.7745270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknd4c3f5/_remote_module_non_sriptable.py 2022-03-17T19:37:54.7785673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpioo9aik5 2022-03-17T19:37:54.7788306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpioo9aik5/_remote_module_non_sriptable.py 2022-03-17T19:37:55.0374974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:37:55.0862518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:37:55.1182951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:37:55.1251936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:00.7248558Z ok (8.402s) 2022-03-17T19:38:00.7248820Z 2022-03-17T19:38:00.7249254Z ---------------------------------------------------------------------- 2022-03-17T19:38:00.7249593Z Ran 1 test in 8.402s 2022-03-17T19:38:00.7249648Z 2022-03-17T19:38:00.7249784Z OK 2022-03-17T19:38:00.7249939Z 2022-03-17T19:38:00.7249992Z Generating XML reports... 2022-03-17T19:38:00.7293391Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193752.xml 2022-03-17T19:38:01.7776407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lklrrk5 2022-03-17T19:38:01.7777443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lklrrk5/_remote_module_non_sriptable.py 2022-03-17T19:38:02.1310188Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:02.1324341Z 2022-03-17T19:38:02.1324627Z Running tests... 2022-03-17T19:38:02.1325057Z ---------------------------------------------------------------------- 2022-03-17T19:38:03.6620184Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:03.7044043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50509 2022-03-17T19:38:03.7169475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50510 2022-03-17T19:38:03.7293204Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50511 2022-03-17T19:38:03.7423897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50512 2022-03-17T19:38:04.5154758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3hs9p7d 2022-03-17T19:38:04.5156628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3hs9p7d/_remote_module_non_sriptable.py 2022-03-17T19:38:04.5198920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7zh1ugv 2022-03-17T19:38:04.5201688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7zh1ugv/_remote_module_non_sriptable.py 2022-03-17T19:38:04.5308081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l_tjxmd 2022-03-17T19:38:04.5310621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l_tjxmd/_remote_module_non_sriptable.py 2022-03-17T19:38:04.5525405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2cr0xxm 2022-03-17T19:38:04.5527591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2cr0xxm/_remote_module_non_sriptable.py 2022-03-17T19:38:04.8632012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:04.8719503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:04.8746940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:04.9016079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:10.5559848Z ok (8.423s) 2022-03-17T19:38:10.5560121Z 2022-03-17T19:38:10.5560943Z ---------------------------------------------------------------------- 2022-03-17T19:38:10.5561290Z Ran 1 test in 8.423s 2022-03-17T19:38:10.5561435Z 2022-03-17T19:38:10.5561537Z OK 2022-03-17T19:38:10.5561658Z 2022-03-17T19:38:10.5561735Z Generating XML reports... 2022-03-17T19:38:10.5604767Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193802.xml 2022-03-17T19:38:11.6116823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_ym4u17 2022-03-17T19:38:11.6117460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_ym4u17/_remote_module_non_sriptable.py 2022-03-17T19:38:11.9639522Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:11.9654825Z 2022-03-17T19:38:11.9655067Z Running tests... 2022-03-17T19:38:11.9655520Z ---------------------------------------------------------------------- 2022-03-17T19:38:13.4888506Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:13.5302151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50754 2022-03-17T19:38:13.5422411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50755 2022-03-17T19:38:13.5544342Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50756 2022-03-17T19:38:13.5666805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50757 2022-03-17T19:38:14.4090290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66n0hg_k 2022-03-17T19:38:14.4092130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66n0hg_k/_remote_module_non_sriptable.py 2022-03-17T19:38:14.4310281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp175wvpi3 2022-03-17T19:38:14.4312992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp175wvpi3/_remote_module_non_sriptable.py 2022-03-17T19:38:14.4376258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpv969ou 2022-03-17T19:38:14.4379441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpv969ou/_remote_module_non_sriptable.py 2022-03-17T19:38:14.4389839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqo8nh_oq 2022-03-17T19:38:14.4392534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqo8nh_oq/_remote_module_non_sriptable.py 2022-03-17T19:38:14.7555165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:14.7783372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:14.7862058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:14.7938799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:20.4834756Z ok (8.518s) 2022-03-17T19:38:20.4834982Z 2022-03-17T19:38:20.4835381Z ---------------------------------------------------------------------- 2022-03-17T19:38:20.4835728Z Ran 1 test in 8.518s 2022-03-17T19:38:20.4835902Z 2022-03-17T19:38:20.4835997Z OK 2022-03-17T19:38:20.4836137Z 2022-03-17T19:38:20.4836273Z Generating XML reports... 2022-03-17T19:38:20.4880911Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193811.xml 2022-03-17T19:38:21.5435158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx61fqa_8 2022-03-17T19:38:21.5436130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx61fqa_8/_remote_module_non_sriptable.py 2022-03-17T19:38:21.8979854Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:21.8995334Z 2022-03-17T19:38:21.8995496Z Running tests... 2022-03-17T19:38:21.8996079Z ---------------------------------------------------------------------- 2022-03-17T19:38:23.4464105Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:23.4885360Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50939 2022-03-17T19:38:23.5006085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50940 2022-03-17T19:38:23.5130832Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50941 2022-03-17T19:38:23.5255350Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50942 2022-03-17T19:38:24.2946157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfty9fjf 2022-03-17T19:38:24.2947137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfty9fjf/_remote_module_non_sriptable.py 2022-03-17T19:38:24.3149141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml8x9m2h 2022-03-17T19:38:24.3151871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml8x9m2h/_remote_module_non_sriptable.py 2022-03-17T19:38:24.3537008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojw3rw94 2022-03-17T19:38:24.3539386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojw3rw94/_remote_module_non_sriptable.py 2022-03-17T19:38:24.3561488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7crmdx4m 2022-03-17T19:38:24.3564266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7crmdx4m/_remote_module_non_sriptable.py 2022-03-17T19:38:24.6408718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:24.6681976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:24.7012010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:24.7048534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:24.9302702Z ok (3.030s) 2022-03-17T19:38:24.9302854Z 2022-03-17T19:38:24.9303302Z ---------------------------------------------------------------------- 2022-03-17T19:38:24.9303670Z Ran 1 test in 3.031s 2022-03-17T19:38:24.9303848Z 2022-03-17T19:38:24.9303931Z OK 2022-03-17T19:38:24.9304069Z 2022-03-17T19:38:24.9304145Z Generating XML reports... 2022-03-17T19:38:24.9351031Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193821.xml 2022-03-17T19:38:25.9892012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7w3saig 2022-03-17T19:38:25.9893634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7w3saig/_remote_module_non_sriptable.py 2022-03-17T19:38:26.3500086Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:26.3515334Z 2022-03-17T19:38:26.3515759Z Running tests... 2022-03-17T19:38:26.3516262Z ---------------------------------------------------------------------- 2022-03-17T19:38:27.8953036Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:27.9378699Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51120 2022-03-17T19:38:27.9498613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51121 2022-03-17T19:38:27.9627418Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51122 2022-03-17T19:38:27.9754134Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51123 2022-03-17T19:38:28.7525961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5urrnhfd 2022-03-17T19:38:28.7526894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5urrnhfd/_remote_module_non_sriptable.py 2022-03-17T19:38:28.7824265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ms79dp7 2022-03-17T19:38:28.7827160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ms79dp7/_remote_module_non_sriptable.py 2022-03-17T19:38:28.8109203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr98icdvw 2022-03-17T19:38:28.8112159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr98icdvw/_remote_module_non_sriptable.py 2022-03-17T19:38:28.8158055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpne2ucjry 2022-03-17T19:38:28.8160667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpne2ucjry/_remote_module_non_sriptable.py 2022-03-17T19:38:29.1086807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:29.1357107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:29.1583704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:29.1690095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:29.3804993Z ok (3.029s) 2022-03-17T19:38:29.3805182Z 2022-03-17T19:38:29.3805603Z ---------------------------------------------------------------------- 2022-03-17T19:38:29.3805921Z Ran 1 test in 3.029s 2022-03-17T19:38:29.3806062Z 2022-03-17T19:38:29.3806162Z OK 2022-03-17T19:38:29.3806283Z 2022-03-17T19:38:29.3806396Z Generating XML reports... 2022-03-17T19:38:29.3849153Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193826.xml 2022-03-17T19:38:30.4339518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptabo5byg 2022-03-17T19:38:30.4341397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptabo5byg/_remote_module_non_sriptable.py 2022-03-17T19:38:30.7901429Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:30.7916611Z 2022-03-17T19:38:30.7916967Z Running tests... 2022-03-17T19:38:30.7917404Z ---------------------------------------------------------------------- 2022-03-17T19:38:32.3365991Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:32.3788872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51301 2022-03-17T19:38:32.3912677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51302 2022-03-17T19:38:32.4043489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51303 2022-03-17T19:38:32.4169410Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51304 2022-03-17T19:38:33.1969116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1z1ec5b9 2022-03-17T19:38:33.1970670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1z1ec5b9/_remote_module_non_sriptable.py 2022-03-17T19:38:33.1975601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelybaony 2022-03-17T19:38:33.1979046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelybaony/_remote_module_non_sriptable.py 2022-03-17T19:38:33.2023211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgy4ongqk 2022-03-17T19:38:33.2026095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgy4ongqk/_remote_module_non_sriptable.py 2022-03-17T19:38:33.2593834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnmclm9q 2022-03-17T19:38:33.2595898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnmclm9q/_remote_module_non_sriptable.py 2022-03-17T19:38:33.5491578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:33.5502634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:33.5559412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:33.6062547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:33.8216851Z ok (3.030s) 2022-03-17T19:38:33.8217076Z 2022-03-17T19:38:33.8217657Z ---------------------------------------------------------------------- 2022-03-17T19:38:33.8218058Z Ran 1 test in 3.030s 2022-03-17T19:38:33.8218192Z 2022-03-17T19:38:33.8218235Z OK 2022-03-17T19:38:33.8218371Z 2022-03-17T19:38:33.8218572Z Generating XML reports... 2022-03-17T19:38:33.8261283Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193830.xml 2022-03-17T19:38:34.8705950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvc5vkfau 2022-03-17T19:38:34.8707201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvc5vkfau/_remote_module_non_sriptable.py 2022-03-17T19:38:35.2266776Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:35.2283403Z 2022-03-17T19:38:35.2283737Z Running tests... 2022-03-17T19:38:35.2284215Z ---------------------------------------------------------------------- 2022-03-17T19:38:36.7706118Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:36.8128609Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51482 2022-03-17T19:38:36.8251858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51483 2022-03-17T19:38:36.8381686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51484 2022-03-17T19:38:36.8508711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51485 2022-03-17T19:38:37.6969321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxum2xipv 2022-03-17T19:38:37.6970270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxum2xipv/_remote_module_non_sriptable.py 2022-03-17T19:38:37.7186889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrxxsotb 2022-03-17T19:38:37.7189631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrxxsotb/_remote_module_non_sriptable.py 2022-03-17T19:38:37.7454560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83pi7ubd 2022-03-17T19:38:37.7456990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83pi7ubd/_remote_module_non_sriptable.py 2022-03-17T19:38:37.7744934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfw6qn7nr 2022-03-17T19:38:37.7747337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfw6qn7nr/_remote_module_non_sriptable.py 2022-03-17T19:38:38.0469576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:38.0616695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:38.0987735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:38.1195310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:38.2556354Z ok (3.027s) 2022-03-17T19:38:38.2556522Z 2022-03-17T19:38:38.2556932Z ---------------------------------------------------------------------- 2022-03-17T19:38:38.2557231Z Ran 1 test in 3.027s 2022-03-17T19:38:38.2557399Z 2022-03-17T19:38:38.2557477Z OK 2022-03-17T19:38:38.2559237Z 2022-03-17T19:38:38.2559635Z Generating XML reports... 2022-03-17T19:38:38.2600200Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193835.xml 2022-03-17T19:38:39.2807091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6poo2xih 2022-03-17T19:38:39.2808386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6poo2xih/_remote_module_non_sriptable.py 2022-03-17T19:38:39.6282100Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:39.6296969Z 2022-03-17T19:38:39.6297280Z Running tests... 2022-03-17T19:38:39.6297781Z ---------------------------------------------------------------------- 2022-03-17T19:38:41.1629540Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:41.2055750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51663 2022-03-17T19:38:41.2181474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51664 2022-03-17T19:38:41.2304631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51665 2022-03-17T19:38:41.2441140Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51666 2022-03-17T19:38:42.0739851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6_ew6r1 2022-03-17T19:38:42.0740895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6_ew6r1/_remote_module_non_sriptable.py 2022-03-17T19:38:42.0805761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tchxahx 2022-03-17T19:38:42.0807096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo69w0csb 2022-03-17T19:38:42.0807899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tchxahx/_remote_module_non_sriptable.py 2022-03-17T19:38:42.0809943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo69w0csb/_remote_module_non_sriptable.py 2022-03-17T19:38:42.0847916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwr6k2c9a 2022-03-17T19:38:42.0851033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwr6k2c9a/_remote_module_non_sriptable.py 2022-03-17T19:38:42.4220094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:42.4253985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:42.4262109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:42.4302299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:44.2517744Z ok (4.622s) 2022-03-17T19:38:44.2518062Z 2022-03-17T19:38:44.2518438Z ---------------------------------------------------------------------- 2022-03-17T19:38:44.2518777Z Ran 1 test in 4.622s 2022-03-17T19:38:44.2518839Z 2022-03-17T19:38:44.2518908Z OK 2022-03-17T19:38:44.2519024Z 2022-03-17T19:38:44.2519158Z Generating XML reports... 2022-03-17T19:38:44.2562080Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193839.xml 2022-03-17T19:38:45.3071789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ps1dqhv 2022-03-17T19:38:45.3073096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ps1dqhv/_remote_module_non_sriptable.py 2022-03-17T19:38:45.6625486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:45.6640563Z 2022-03-17T19:38:45.6640955Z Running tests... 2022-03-17T19:38:45.6641445Z ---------------------------------------------------------------------- 2022-03-17T19:38:47.1915133Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:47.2333142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51848 2022-03-17T19:38:47.2453914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51849 2022-03-17T19:38:47.2577190Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51850 2022-03-17T19:38:47.2701958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51851 2022-03-17T19:38:48.0708927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo99ns9dd 2022-03-17T19:38:48.0710427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo99ns9dd/_remote_module_non_sriptable.py 2022-03-17T19:38:48.0812051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzsonziu 2022-03-17T19:38:48.0814767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzsonziu/_remote_module_non_sriptable.py 2022-03-17T19:38:48.0819261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtb40dvu 2022-03-17T19:38:48.0822271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtb40dvu/_remote_module_non_sriptable.py 2022-03-17T19:38:48.1412352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejktpg0t 2022-03-17T19:38:48.1414496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejktpg0t/_remote_module_non_sriptable.py 2022-03-17T19:38:48.4219990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:48.4233835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:48.4283510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:38:48.4941728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:50.2780739Z ok (4.614s) 2022-03-17T19:38:50.2781324Z 2022-03-17T19:38:50.2781751Z ---------------------------------------------------------------------- 2022-03-17T19:38:50.2782081Z Ran 1 test in 4.614s 2022-03-17T19:38:50.2782175Z 2022-03-17T19:38:50.2782575Z OK 2022-03-17T19:38:50.2782733Z 2022-03-17T19:38:50.2782855Z Generating XML reports... 2022-03-17T19:38:50.2827057Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193845.xml 2022-03-17T19:38:51.3241929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wv1o2o3 2022-03-17T19:38:51.3242838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wv1o2o3/_remote_module_non_sriptable.py 2022-03-17T19:38:51.6750948Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:38:51.6766058Z 2022-03-17T19:38:51.6766554Z Running tests... 2022-03-17T19:38:51.6767080Z ---------------------------------------------------------------------- 2022-03-17T19:38:53.1931547Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:38:53.2350054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52033 2022-03-17T19:38:53.2470632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52034 2022-03-17T19:38:53.2591890Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52035 2022-03-17T19:38:53.2714822Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52036 2022-03-17T19:38:54.0567549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptev40x3r 2022-03-17T19:38:54.0568485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u08_j_t 2022-03-17T19:38:54.0569645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptev40x3r/_remote_module_non_sriptable.py 2022-03-17T19:38:54.0570170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u08_j_t/_remote_module_non_sriptable.py 2022-03-17T19:38:54.0637003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33c63kk4 2022-03-17T19:38:54.0639561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33c63kk4/_remote_module_non_sriptable.py 2022-03-17T19:38:54.0856348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnv86v71 2022-03-17T19:38:54.0859237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnv86v71/_remote_module_non_sriptable.py 2022-03-17T19:38:54.3990186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:38:54.4058503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:38:54.4170729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:38:54.4373441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:39:01.3879166Z ok (9.711s) 2022-03-17T19:39:01.3879363Z 2022-03-17T19:39:01.3879800Z ---------------------------------------------------------------------- 2022-03-17T19:39:01.3880139Z Ran 1 test in 9.711s 2022-03-17T19:39:01.3880305Z 2022-03-17T19:39:01.3880379Z OK 2022-03-17T19:39:01.3880513Z 2022-03-17T19:39:01.3880646Z Generating XML reports... 2022-03-17T19:39:01.3923504Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193851.xml 2022-03-17T19:39:02.4516090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyf9ck_57 2022-03-17T19:39:02.4517276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyf9ck_57/_remote_module_non_sriptable.py 2022-03-17T19:39:02.8062862Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:39:02.8078501Z 2022-03-17T19:39:02.8078840Z Running tests... 2022-03-17T19:39:02.8079381Z ---------------------------------------------------------------------- 2022-03-17T19:39:04.3305150Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:39:04.3722978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52222 2022-03-17T19:39:04.3844291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52223 2022-03-17T19:39:04.3967568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52224 2022-03-17T19:39:04.4092567Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52225 2022-03-17T19:39:05.1933144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_0cwhln 2022-03-17T19:39:05.1934168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_0cwhln/_remote_module_non_sriptable.py 2022-03-17T19:39:05.2134839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0solg9mg 2022-03-17T19:39:05.2138560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0solg9mg/_remote_module_non_sriptable.py 2022-03-17T19:39:05.2221338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fliage5 2022-03-17T19:39:05.2224717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fliage5/_remote_module_non_sriptable.py 2022-03-17T19:39:05.2398599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6h_9ekp 2022-03-17T19:39:05.2401464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6h_9ekp/_remote_module_non_sriptable.py 2022-03-17T19:39:05.5370916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:39:05.5635184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:05.5774295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:39:05.5892087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:13.3282208Z ok (10.520s) 2022-03-17T19:39:13.3282540Z 2022-03-17T19:39:13.3282966Z ---------------------------------------------------------------------- 2022-03-17T19:39:13.3283272Z Ran 1 test in 10.520s 2022-03-17T19:39:13.3283372Z 2022-03-17T19:39:13.3283467Z OK 2022-03-17T19:39:13.3283606Z 2022-03-17T19:39:13.3283755Z Generating XML reports... 2022-03-17T19:39:13.3325810Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193902.xml 2022-03-17T19:39:14.3703423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfqclwnq 2022-03-17T19:39:14.3704757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfqclwnq/_remote_module_non_sriptable.py 2022-03-17T19:39:14.7277080Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:39:14.7292865Z 2022-03-17T19:39:14.7293110Z Running tests... 2022-03-17T19:39:14.7293558Z ---------------------------------------------------------------------- 2022-03-17T19:39:16.2784777Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:39:16.3204496Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52587 2022-03-17T19:39:16.3324482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52588 2022-03-17T19:39:16.3450494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52589 2022-03-17T19:39:16.3573001Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52590 2022-03-17T19:39:17.1321440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45n0yunp 2022-03-17T19:39:17.1322535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78r0edqt 2022-03-17T19:39:17.1323587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45n0yunp/_remote_module_non_sriptable.py 2022-03-17T19:39:17.1325143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78r0edqt/_remote_module_non_sriptable.py 2022-03-17T19:39:17.1868006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7zn4g6i 2022-03-17T19:39:17.1870103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7zn4g6i/_remote_module_non_sriptable.py 2022-03-17T19:39:17.1925424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphknynba1 2022-03-17T19:39:17.1929127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphknynba1/_remote_module_non_sriptable.py 2022-03-17T19:39:17.4793684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:39:17.4839146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:17.5301591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:39:17.5412908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:31.2849805Z ok (16.555s) 2022-03-17T19:39:31.2850055Z 2022-03-17T19:39:31.2850443Z ---------------------------------------------------------------------- 2022-03-17T19:39:31.2850786Z Ran 1 test in 16.556s 2022-03-17T19:39:31.2850970Z 2022-03-17T19:39:31.2851062Z OK 2022-03-17T19:39:31.2851180Z 2022-03-17T19:39:31.2851316Z Generating XML reports... 2022-03-17T19:39:31.2894508Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193914.xml 2022-03-17T19:39:32.2946350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1vflhli 2022-03-17T19:39:32.2947597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1vflhli/_remote_module_non_sriptable.py 2022-03-17T19:39:32.6475430Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:39:32.6491925Z 2022-03-17T19:39:32.6492363Z Running tests... 2022-03-17T19:39:32.6493152Z ---------------------------------------------------------------------- 2022-03-17T19:39:34.1987711Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:39:34.2408506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52952 2022-03-17T19:39:34.2530304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52953 2022-03-17T19:39:34.2657390Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52954 2022-03-17T19:39:34.2779253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52955 2022-03-17T19:39:35.0692290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1pwttig 2022-03-17T19:39:35.0693333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1pwttig/_remote_module_non_sriptable.py 2022-03-17T19:39:35.1067893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgj5eyzak 2022-03-17T19:39:35.1070261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgj5eyzak/_remote_module_non_sriptable.py 2022-03-17T19:39:35.1268895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4wfiz3a 2022-03-17T19:39:35.1271649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4wfiz3a/_remote_module_non_sriptable.py 2022-03-17T19:39:35.1507175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacxsuqps 2022-03-17T19:39:35.1509983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacxsuqps/_remote_module_non_sriptable.py 2022-03-17T19:39:35.4315809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:35.4532364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:39:35.4739030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:35.4941113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:39:44.3987145Z ok (11.749s) 2022-03-17T19:39:44.3987352Z 2022-03-17T19:39:44.3987752Z ---------------------------------------------------------------------- 2022-03-17T19:39:44.3988115Z Ran 1 test in 11.749s 2022-03-17T19:39:44.3988281Z 2022-03-17T19:39:44.3988353Z OK 2022-03-17T19:39:44.3988491Z 2022-03-17T19:39:44.3988628Z Generating XML reports... 2022-03-17T19:39:44.4032805Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193932.xml 2022-03-17T19:39:45.4350640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8if_uu22 2022-03-17T19:39:45.4351429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8if_uu22/_remote_module_non_sriptable.py 2022-03-17T19:39:45.7800773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:39:45.7815318Z 2022-03-17T19:39:45.7815561Z Running tests... 2022-03-17T19:39:45.7815998Z ---------------------------------------------------------------------- 2022-03-17T19:39:47.2787525Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:39:47.3196215Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53317 2022-03-17T19:39:47.3315255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53318 2022-03-17T19:39:47.3436234Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53319 2022-03-17T19:39:47.3563600Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53320 2022-03-17T19:39:48.1672703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7n_swu3q 2022-03-17T19:39:48.1673582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7n_swu3q/_remote_module_non_sriptable.py 2022-03-17T19:39:48.1676434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx97yz0mz 2022-03-17T19:39:48.1679337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx97yz0mz/_remote_module_non_sriptable.py 2022-03-17T19:39:48.2147329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ig3uspi 2022-03-17T19:39:48.2149632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ig3uspi/_remote_module_non_sriptable.py 2022-03-17T19:39:48.2439125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqg0hom96 2022-03-17T19:39:48.2441940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqg0hom96/_remote_module_non_sriptable.py 2022-03-17T19:39:48.5122088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:39:48.5129467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:39:48.5634557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:39:48.5956350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:39:55.7736950Z ok (9.992s) 2022-03-17T19:39:55.7737167Z 2022-03-17T19:39:55.7737991Z ---------------------------------------------------------------------- 2022-03-17T19:39:55.7738347Z Ran 1 test in 9.992s 2022-03-17T19:39:55.7738515Z 2022-03-17T19:39:55.7738617Z OK 2022-03-17T19:39:55.7738708Z 2022-03-17T19:39:55.7738841Z Generating XML reports... 2022-03-17T19:39:55.7781754Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193945.xml 2022-03-17T19:39:56.8138051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6883jz32 2022-03-17T19:39:56.8140362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6883jz32/_remote_module_non_sriptable.py 2022-03-17T19:39:57.1640285Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:39:57.1655540Z 2022-03-17T19:39:57.1655939Z Running tests... 2022-03-17T19:39:57.1656376Z ---------------------------------------------------------------------- 2022-03-17T19:39:58.7030403Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:39:58.7449882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53677 2022-03-17T19:39:58.7569893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53678 2022-03-17T19:39:58.7691788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53679 2022-03-17T19:39:58.7820986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53680 2022-03-17T19:39:59.6327758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2h9byaq5 2022-03-17T19:39:59.6328932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2h9byaq5/_remote_module_non_sriptable.py 2022-03-17T19:39:59.6560295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuaoih9wp 2022-03-17T19:39:59.6562639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuaoih9wp/_remote_module_non_sriptable.py 2022-03-17T19:39:59.6564122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_d5mqvl1 2022-03-17T19:39:59.6567548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_d5mqvl1/_remote_module_non_sriptable.py 2022-03-17T19:39:59.6586398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5rhe9_7 2022-03-17T19:39:59.6589648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5rhe9_7/_remote_module_non_sriptable.py 2022-03-17T19:39:59.9877625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:39:59.9986801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:00.0010291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:40:00.0097563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:00.6877366Z ok (3.522s) 2022-03-17T19:40:00.6877599Z 2022-03-17T19:40:00.6878008Z ---------------------------------------------------------------------- 2022-03-17T19:40:00.6878324Z Ran 1 test in 3.522s 2022-03-17T19:40:00.6878489Z 2022-03-17T19:40:00.6878568Z OK 2022-03-17T19:40:00.6878642Z 2022-03-17T19:40:00.6878766Z Generating XML reports... 2022-03-17T19:40:00.6921945Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193957.xml 2022-03-17T19:40:01.7413649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5dpne7d 2022-03-17T19:40:01.7414832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5dpne7d/_remote_module_non_sriptable.py 2022-03-17T19:40:02.0951793Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:40:02.0969951Z 2022-03-17T19:40:02.0970323Z Running tests... 2022-03-17T19:40:02.0970822Z ---------------------------------------------------------------------- 2022-03-17T19:40:03.6238634Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:40:03.6657741Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54026 2022-03-17T19:40:03.6782434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54027 2022-03-17T19:40:03.6907291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54028 2022-03-17T19:40:03.7034130Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54029 2022-03-17T19:40:04.5250250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmiwmtjd 2022-03-17T19:40:04.5250908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmiwmtjd/_remote_module_non_sriptable.py 2022-03-17T19:40:04.5424591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mrkkl30 2022-03-17T19:40:04.5427784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mrkkl30/_remote_module_non_sriptable.py 2022-03-17T19:40:04.5553749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm6yl89d 2022-03-17T19:40:04.5556140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm6yl89d/_remote_module_non_sriptable.py 2022-03-17T19:40:04.5575309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81h2_8tm 2022-03-17T19:40:04.5578153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81h2_8tm/_remote_module_non_sriptable.py 2022-03-17T19:40:04.8820569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:04.8904106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:04.8987016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:40:04.9013567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:40:08.3142522Z ok (6.217s) 2022-03-17T19:40:08.3142761Z 2022-03-17T19:40:08.3143153Z ---------------------------------------------------------------------- 2022-03-17T19:40:08.3143493Z Ran 1 test in 6.217s 2022-03-17T19:40:08.3143605Z 2022-03-17T19:40:08.3143743Z OK 2022-03-17T19:40:08.3143871Z 2022-03-17T19:40:08.3143951Z Generating XML reports... 2022-03-17T19:40:08.3188379Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194002.xml 2022-03-17T19:40:09.3588968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1203j5lg 2022-03-17T19:40:09.3589763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1203j5lg/_remote_module_non_sriptable.py 2022-03-17T19:40:09.7099272Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:40:09.7114985Z 2022-03-17T19:40:09.7115275Z Running tests... 2022-03-17T19:40:09.7115686Z ---------------------------------------------------------------------- 2022-03-17T19:40:11.2613480Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:40:11.3031037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54383 2022-03-17T19:40:11.3151931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54384 2022-03-17T19:40:11.3278254Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54385 2022-03-17T19:40:11.3402197Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54386 2022-03-17T19:40:12.1776126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvqx1_ik 2022-03-17T19:40:12.1777523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvqx1_ik/_remote_module_non_sriptable.py 2022-03-17T19:40:12.2035360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74q_lmkv 2022-03-17T19:40:12.2037245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74q_lmkv/_remote_module_non_sriptable.py 2022-03-17T19:40:12.2116451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43xf37_t 2022-03-17T19:40:12.2120039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43xf37_t/_remote_module_non_sriptable.py 2022-03-17T19:40:12.2340171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk58pxla0 2022-03-17T19:40:12.2342184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk58pxla0/_remote_module_non_sriptable.py 2022-03-17T19:40:12.5339687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:12.5475155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:40:12.5640236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:12.5784838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:40:15.9518368Z ok (6.240s) 2022-03-17T19:40:15.9518845Z 2022-03-17T19:40:15.9519437Z ---------------------------------------------------------------------- 2022-03-17T19:40:15.9520034Z Ran 1 test in 6.240s 2022-03-17T19:40:15.9520331Z 2022-03-17T19:40:15.9520488Z OK 2022-03-17T19:40:15.9520735Z 2022-03-17T19:40:15.9520967Z Generating XML reports... 2022-03-17T19:40:15.9566491Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194009.xml 2022-03-17T19:40:16.9996692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnx_du8ws 2022-03-17T19:40:16.9998073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnx_du8ws/_remote_module_non_sriptable.py 2022-03-17T19:40:17.3621996Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:40:17.3638219Z 2022-03-17T19:40:17.3638668Z Running tests... 2022-03-17T19:40:17.3639171Z ---------------------------------------------------------------------- 2022-03-17T19:40:18.8996266Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:40:18.9415079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54740 2022-03-17T19:40:18.9534358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54741 2022-03-17T19:40:18.9658659Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54742 2022-03-17T19:40:18.9785879Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54743 2022-03-17T19:40:19.7561234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5d07p5pr 2022-03-17T19:40:19.7562231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5d07p5pr/_remote_module_non_sriptable.py 2022-03-17T19:40:19.8099670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gwy2_vp 2022-03-17T19:40:19.8101022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gwy2_vp/_remote_module_non_sriptable.py 2022-03-17T19:40:19.8258743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6apmtbxt 2022-03-17T19:40:19.8261126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6apmtbxt/_remote_module_non_sriptable.py 2022-03-17T19:40:19.8483197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuft7b9hq 2022-03-17T19:40:19.8485788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuft7b9hq/_remote_module_non_sriptable.py 2022-03-17T19:40:20.1053028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:20.1513651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:40:20.1740392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:20.1981081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:40:23.5897984Z ok (6.226s) 2022-03-17T19:40:23.5898216Z 2022-03-17T19:40:23.5898615Z ---------------------------------------------------------------------- 2022-03-17T19:40:23.5898935Z Ran 1 test in 6.226s 2022-03-17T19:40:23.5899084Z 2022-03-17T19:40:23.5899177Z OK 2022-03-17T19:40:23.5899301Z 2022-03-17T19:40:23.5899719Z Generating XML reports... 2022-03-17T19:40:23.5942442Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194017.xml 2022-03-17T19:40:24.6310251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyxb3t84 2022-03-17T19:40:24.6311057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyxb3t84/_remote_module_non_sriptable.py 2022-03-17T19:40:24.9858779Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:40:24.9876501Z 2022-03-17T19:40:24.9876752Z Running tests... 2022-03-17T19:40:24.9877386Z ---------------------------------------------------------------------- 2022-03-17T19:40:26.5224466Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:40:26.5643869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55093 2022-03-17T19:40:26.5762353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55094 2022-03-17T19:40:26.5885618Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55095 2022-03-17T19:40:26.6012984Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55096 2022-03-17T19:40:27.3636185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9b0fcnk 2022-03-17T19:40:27.3637401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9b0fcnk/_remote_module_non_sriptable.py 2022-03-17T19:40:27.3812663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdzfd6xf 2022-03-17T19:40:27.3815357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdzfd6xf/_remote_module_non_sriptable.py 2022-03-17T19:40:27.3882602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperh2lkn_ 2022-03-17T19:40:27.3885193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperh2lkn_/_remote_module_non_sriptable.py 2022-03-17T19:40:27.4286246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0adgbiwu 2022-03-17T19:40:27.4288636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0adgbiwu/_remote_module_non_sriptable.py 2022-03-17T19:40:27.7125548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:27.7256653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:40:27.7333821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:40:27.7751021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:33.5162534Z ok (8.528s) 2022-03-17T19:40:33.5162909Z 2022-03-17T19:40:33.5163478Z ---------------------------------------------------------------------- 2022-03-17T19:40:33.5163854Z Ran 1 test in 8.529s 2022-03-17T19:40:33.5164024Z 2022-03-17T19:40:33.5164097Z OK 2022-03-17T19:40:33.5164271Z 2022-03-17T19:40:33.5164522Z Generating XML reports... 2022-03-17T19:40:33.5208034Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194024.xml 2022-03-17T19:40:34.5475468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_fsoz4w2 2022-03-17T19:40:34.5476588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_fsoz4w2/_remote_module_non_sriptable.py 2022-03-17T19:40:34.8939212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:40:34.8955467Z 2022-03-17T19:40:34.8955565Z Running tests... 2022-03-17T19:40:34.8956345Z ---------------------------------------------------------------------- 2022-03-17T19:40:36.4101264Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:40:36.4513715Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55458 2022-03-17T19:40:36.4632882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55459 2022-03-17T19:40:36.4754430Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55460 2022-03-17T19:40:36.4876286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55461 2022-03-17T19:40:37.3300998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpae5ymx23 2022-03-17T19:40:37.3301879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpae5ymx23/_remote_module_non_sriptable.py 2022-03-17T19:40:37.3513484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsysi9g02 2022-03-17T19:40:37.3516232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsysi9g02/_remote_module_non_sriptable.py 2022-03-17T19:40:37.3722182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcdms8g1 2022-03-17T19:40:37.3725113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcdms8g1/_remote_module_non_sriptable.py 2022-03-17T19:40:37.3764187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c7umhb7 2022-03-17T19:40:37.3767072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c7umhb7/_remote_module_non_sriptable.py 2022-03-17T19:40:37.6743935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:37.6970831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:37.7265352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:40:37.7266264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:40:43.4025658Z ok (8.507s) 2022-03-17T19:40:43.4025903Z 2022-03-17T19:40:43.4026305Z ---------------------------------------------------------------------- 2022-03-17T19:40:43.4026625Z Ran 1 test in 8.507s 2022-03-17T19:40:43.4026791Z 2022-03-17T19:40:43.4026884Z OK 2022-03-17T19:40:43.4027885Z 2022-03-17T19:40:43.4028017Z Generating XML reports... 2022-03-17T19:40:43.4070458Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194034.xml 2022-03-17T19:40:44.4444490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04xp3i6l 2022-03-17T19:40:44.4445723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04xp3i6l/_remote_module_non_sriptable.py 2022-03-17T19:40:44.8001482Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:40:44.8017109Z 2022-03-17T19:40:44.8017472Z Running tests... 2022-03-17T19:40:44.8018026Z ---------------------------------------------------------------------- 2022-03-17T19:40:46.3211154Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:40:46.3633230Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55815 2022-03-17T19:40:46.3756436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55816 2022-03-17T19:40:46.3880646Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55817 2022-03-17T19:40:46.4008466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55818 2022-03-17T19:40:47.1929538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_i07poyv 2022-03-17T19:40:47.1930711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_i07poyv/_remote_module_non_sriptable.py 2022-03-17T19:40:47.2127410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsaqpnb0p 2022-03-17T19:40:47.2130541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsaqpnb0p/_remote_module_non_sriptable.py 2022-03-17T19:40:47.2703941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps93p3oi5 2022-03-17T19:40:47.2706173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps93p3oi5/_remote_module_non_sriptable.py 2022-03-17T19:40:47.2756758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8v4d0fe 2022-03-17T19:40:47.2759576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8v4d0fe/_remote_module_non_sriptable.py 2022-03-17T19:40:47.5437418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:47.5590757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:47.6152146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:40:47.6281464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:40:53.3164057Z ok (8.514s) 2022-03-17T19:40:53.3164232Z 2022-03-17T19:40:53.3164701Z ---------------------------------------------------------------------- 2022-03-17T19:40:53.3165254Z Ran 1 test in 8.515s 2022-03-17T19:40:53.3165407Z 2022-03-17T19:40:53.3165524Z OK 2022-03-17T19:40:53.3165661Z 2022-03-17T19:40:53.3165802Z Generating XML reports... 2022-03-17T19:40:53.3209422Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194044.xml 2022-03-17T19:40:54.3581081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mu09kpa 2022-03-17T19:40:54.3582266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mu09kpa/_remote_module_non_sriptable.py 2022-03-17T19:40:54.7091908Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:40:54.7107829Z 2022-03-17T19:40:54.7108199Z Running tests... 2022-03-17T19:40:54.7108624Z ---------------------------------------------------------------------- 2022-03-17T19:40:56.2223576Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:40:56.2636698Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56172 2022-03-17T19:40:56.2757302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56173 2022-03-17T19:40:56.2879173Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56174 2022-03-17T19:40:56.3002825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56175 2022-03-17T19:40:57.1297571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkbplwy2 2022-03-17T19:40:57.1298168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkbplwy2/_remote_module_non_sriptable.py 2022-03-17T19:40:57.1462490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzz050ge 2022-03-17T19:40:57.1465379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzz050ge/_remote_module_non_sriptable.py 2022-03-17T19:40:57.1646440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8u6seap 2022-03-17T19:40:57.1649216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8u6seap/_remote_module_non_sriptable.py 2022-03-17T19:40:57.1680457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpap3st5rs 2022-03-17T19:40:57.1683793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpap3st5rs/_remote_module_non_sriptable.py 2022-03-17T19:40:57.4762223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:40:57.4946256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:40:57.5142238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:40:57.5203432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:03.2154098Z ok (8.504s) 2022-03-17T19:41:03.2154234Z 2022-03-17T19:41:03.2154609Z ---------------------------------------------------------------------- 2022-03-17T19:41:03.2155272Z Ran 1 test in 8.505s 2022-03-17T19:41:03.2155475Z 2022-03-17T19:41:03.2155589Z OK 2022-03-17T19:41:03.2155838Z 2022-03-17T19:41:03.2156074Z Generating XML reports... 2022-03-17T19:41:03.2198540Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194054.xml 2022-03-17T19:41:04.2605971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybda64l9 2022-03-17T19:41:04.2606811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybda64l9/_remote_module_non_sriptable.py 2022-03-17T19:41:04.6218757Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:41:04.6234711Z 2022-03-17T19:41:04.6234925Z Running tests... 2022-03-17T19:41:04.6235415Z ---------------------------------------------------------------------- 2022-03-17T19:41:06.1754368Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:41:06.2173827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56529 2022-03-17T19:41:06.2294965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56530 2022-03-17T19:41:06.2417878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56531 2022-03-17T19:41:06.2542760Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56532 2022-03-17T19:41:07.0276226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0hz36n1 2022-03-17T19:41:07.0277683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0hz36n1/_remote_module_non_sriptable.py 2022-03-17T19:41:07.0342088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2xyhivjs 2022-03-17T19:41:07.0345043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2xyhivjs/_remote_module_non_sriptable.py 2022-03-17T19:41:07.0846793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4m5ticxw 2022-03-17T19:41:07.0849164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4m5ticxw/_remote_module_non_sriptable.py 2022-03-17T19:41:07.1009844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_hr4xoz 2022-03-17T19:41:07.1012654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_hr4xoz/_remote_module_non_sriptable.py 2022-03-17T19:41:07.3793541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:07.3814454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:07.4309296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:07.4506927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:13.0693842Z ok (8.446s) 2022-03-17T19:41:13.0694390Z 2022-03-17T19:41:13.0694762Z ---------------------------------------------------------------------- 2022-03-17T19:41:13.0695090Z Ran 1 test in 8.446s 2022-03-17T19:41:13.0695260Z 2022-03-17T19:41:13.0696293Z OK 2022-03-17T19:41:13.0696432Z 2022-03-17T19:41:13.0696555Z Generating XML reports... 2022-03-17T19:41:13.0738546Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194104.xml 2022-03-17T19:41:14.1072998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiazs_ob 2022-03-17T19:41:14.1074759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiazs_ob/_remote_module_non_sriptable.py 2022-03-17T19:41:14.4618580Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:41:14.4634667Z 2022-03-17T19:41:14.4634957Z Running tests... 2022-03-17T19:41:14.4635424Z ---------------------------------------------------------------------- 2022-03-17T19:41:15.9854045Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:41:16.0272173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56886 2022-03-17T19:41:16.0391801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56887 2022-03-17T19:41:16.0515833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56888 2022-03-17T19:41:16.0640513Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56889 2022-03-17T19:41:16.8514232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1l__61r 2022-03-17T19:41:16.8515020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1l__61r/_remote_module_non_sriptable.py 2022-03-17T19:41:16.8769315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpim7e14mt 2022-03-17T19:41:16.8772212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpim7e14mt/_remote_module_non_sriptable.py 2022-03-17T19:41:16.9047347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3g6x9bng 2022-03-17T19:41:16.9049840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3g6x9bng/_remote_module_non_sriptable.py 2022-03-17T19:41:16.9193941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35tq9vm5 2022-03-17T19:41:16.9196794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35tq9vm5/_remote_module_non_sriptable.py 2022-03-17T19:41:17.2058954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:17.2251711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:17.2507302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:17.2697302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:22.9796968Z ok (8.516s) 2022-03-17T19:41:22.9797197Z 2022-03-17T19:41:22.9797604Z ---------------------------------------------------------------------- 2022-03-17T19:41:22.9797964Z Ran 1 test in 8.516s 2022-03-17T19:41:22.9798121Z 2022-03-17T19:41:22.9798215Z OK 2022-03-17T19:41:22.9798330Z 2022-03-17T19:41:22.9798469Z Generating XML reports... 2022-03-17T19:41:22.9842226Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194114.xml 2022-03-17T19:41:23.9769606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7tuq5iz 2022-03-17T19:41:23.9770803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7tuq5iz/_remote_module_non_sriptable.py 2022-03-17T19:41:24.3361681Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:41:24.3378514Z 2022-03-17T19:41:24.3379098Z Running tests... 2022-03-17T19:41:24.3379901Z ---------------------------------------------------------------------- 2022-03-17T19:41:25.8698595Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:41:25.9110975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57251 2022-03-17T19:41:25.9232095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57252 2022-03-17T19:41:25.9354366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57253 2022-03-17T19:41:25.9480316Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57254 2022-03-17T19:41:26.7248967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__lu1bhj 2022-03-17T19:41:26.7250043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__lu1bhj/_remote_module_non_sriptable.py 2022-03-17T19:41:26.7770037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponrt_6zp 2022-03-17T19:41:26.7771540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponrt_6zp/_remote_module_non_sriptable.py 2022-03-17T19:41:26.7831309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlh5lgx9 2022-03-17T19:41:26.7833430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlh5lgx9/_remote_module_non_sriptable.py 2022-03-17T19:41:26.8006887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfjib6o8 2022-03-17T19:41:26.8008700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfjib6o8/_remote_module_non_sriptable.py 2022-03-17T19:41:27.0785500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:27.1240097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:27.1286593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:27.1467093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:32.8636572Z ok (8.525s) 2022-03-17T19:41:32.8636802Z 2022-03-17T19:41:32.8637453Z ---------------------------------------------------------------------- 2022-03-17T19:41:32.8637788Z Ran 1 test in 8.526s 2022-03-17T19:41:32.8637983Z 2022-03-17T19:41:32.8638048Z OK 2022-03-17T19:41:32.8638142Z 2022-03-17T19:41:32.8638277Z Generating XML reports... 2022-03-17T19:41:32.8682050Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194124.xml 2022-03-17T19:41:33.9045949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus2ajr33 2022-03-17T19:41:33.9047221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus2ajr33/_remote_module_non_sriptable.py 2022-03-17T19:41:34.2652856Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:41:34.2673862Z 2022-03-17T19:41:34.2674358Z Running tests... 2022-03-17T19:41:34.2674788Z ---------------------------------------------------------------------- 2022-03-17T19:41:35.7850030Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:41:35.8266921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57616 2022-03-17T19:41:35.8386643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57617 2022-03-17T19:41:35.8507873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57618 2022-03-17T19:41:35.8631053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57619 2022-03-17T19:41:36.6701619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpte6cs1zl 2022-03-17T19:41:36.6702650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpte6cs1zl/_remote_module_non_sriptable.py 2022-03-17T19:41:36.7044120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehujv76y 2022-03-17T19:41:36.7046804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehujv76y/_remote_module_non_sriptable.py 2022-03-17T19:41:36.7088569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqiz_gdzn 2022-03-17T19:41:36.7091364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqiz_gdzn/_remote_module_non_sriptable.py 2022-03-17T19:41:36.7227837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeo29u3ux 2022-03-17T19:41:36.7230886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeo29u3ux/_remote_module_non_sriptable.py 2022-03-17T19:41:37.0162294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:37.0461963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:37.0507573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:37.0656798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:42.7781861Z ok (8.511s) 2022-03-17T19:41:42.7782142Z 2022-03-17T19:41:42.7782574Z ---------------------------------------------------------------------- 2022-03-17T19:41:42.7782893Z Ran 1 test in 8.511s 2022-03-17T19:41:42.7783070Z 2022-03-17T19:41:42.7785020Z OK 2022-03-17T19:41:42.7785328Z 2022-03-17T19:41:42.7785563Z Generating XML reports... 2022-03-17T19:41:42.7827204Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194134.xml 2022-03-17T19:41:43.8179338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j2u275k 2022-03-17T19:41:43.8180728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j2u275k/_remote_module_non_sriptable.py 2022-03-17T19:41:44.1762824Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:41:44.1778043Z 2022-03-17T19:41:44.1778450Z Running tests... 2022-03-17T19:41:44.1779205Z ---------------------------------------------------------------------- 2022-03-17T19:41:45.7152806Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:41:45.7567223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57981 2022-03-17T19:41:45.7687478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57982 2022-03-17T19:41:45.7809918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57983 2022-03-17T19:41:45.7938014Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57984 2022-03-17T19:41:46.5984134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbt36aep4 2022-03-17T19:41:46.5984996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbt36aep4/_remote_module_non_sriptable.py 2022-03-17T19:41:46.6266588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_ou381u 2022-03-17T19:41:46.6269096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_ou381u/_remote_module_non_sriptable.py 2022-03-17T19:41:46.6317815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9eonmij2 2022-03-17T19:41:46.6321459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9eonmij2/_remote_module_non_sriptable.py 2022-03-17T19:41:46.6747623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghjldqkd 2022-03-17T19:41:46.6750224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghjldqkd/_remote_module_non_sriptable.py 2022-03-17T19:41:46.9537935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:46.9748778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:41:46.9794824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:47.0182570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:52.7091777Z ok (8.531s) 2022-03-17T19:41:52.7092035Z 2022-03-17T19:41:52.7092427Z ---------------------------------------------------------------------- 2022-03-17T19:41:52.7092746Z Ran 1 test in 8.531s 2022-03-17T19:41:52.7092893Z 2022-03-17T19:41:52.7092979Z OK 2022-03-17T19:41:52.7093115Z 2022-03-17T19:41:52.7093231Z Generating XML reports... 2022-03-17T19:41:52.7137684Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194144.xml 2022-03-17T19:41:53.7505507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq62xtnp3 2022-03-17T19:41:53.7506591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq62xtnp3/_remote_module_non_sriptable.py 2022-03-17T19:41:54.1020228Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:41:54.1035910Z 2022-03-17T19:41:54.1036346Z Running tests... 2022-03-17T19:41:54.1036849Z ---------------------------------------------------------------------- 2022-03-17T19:41:55.6478451Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:41:55.6894930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58346 2022-03-17T19:41:55.7017098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58347 2022-03-17T19:41:55.7146389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58348 2022-03-17T19:41:55.7273436Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58349 2022-03-17T19:41:56.5161403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsd60itt 2022-03-17T19:41:56.5162384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsd60itt/_remote_module_non_sriptable.py 2022-03-17T19:41:56.5267121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3y0rjapt 2022-03-17T19:41:56.5269830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3y0rjapt/_remote_module_non_sriptable.py 2022-03-17T19:41:56.5470584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdz3jbtx 2022-03-17T19:41:56.5473295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdz3jbtx/_remote_module_non_sriptable.py 2022-03-17T19:41:56.5586071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zpoorq1 2022-03-17T19:41:56.5589236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zpoorq1/_remote_module_non_sriptable.py 2022-03-17T19:41:56.8628441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:41:56.8730419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:41:56.8879704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:41:56.9046203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:02.6434905Z ok (8.540s) 2022-03-17T19:42:02.6435123Z 2022-03-17T19:42:02.6435556Z ---------------------------------------------------------------------- 2022-03-17T19:42:02.6435925Z Ran 1 test in 8.540s 2022-03-17T19:42:02.6436093Z 2022-03-17T19:42:02.6436218Z OK 2022-03-17T19:42:02.6436341Z 2022-03-17T19:42:02.6436443Z Generating XML reports... 2022-03-17T19:42:02.6481481Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194154.xml 2022-03-17T19:42:03.6875646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gifp5m8 2022-03-17T19:42:03.6876566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gifp5m8/_remote_module_non_sriptable.py 2022-03-17T19:42:04.0425915Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:42:04.0441328Z 2022-03-17T19:42:04.0441567Z Running tests... 2022-03-17T19:42:04.0442034Z ---------------------------------------------------------------------- 2022-03-17T19:42:05.5668033Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:42:05.6089299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58703 2022-03-17T19:42:05.6209802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58704 2022-03-17T19:42:05.6333278Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58705 2022-03-17T19:42:05.6457969Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58706 2022-03-17T19:42:06.4123645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu91n5lz4 2022-03-17T19:42:06.4124730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu91n5lz4/_remote_module_non_sriptable.py 2022-03-17T19:42:06.4366482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsbmqaey 2022-03-17T19:42:06.4368469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsbmqaey/_remote_module_non_sriptable.py 2022-03-17T19:42:06.4620019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2mjvbjr 2022-03-17T19:42:06.4622292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2mjvbjr/_remote_module_non_sriptable.py 2022-03-17T19:42:06.4798044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_9l6677 2022-03-17T19:42:06.4800061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_9l6677/_remote_module_non_sriptable.py 2022-03-17T19:42:06.7600045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:06.7800930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:06.8114130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:06.8279673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:12.4610551Z ok (8.416s) 2022-03-17T19:42:12.4610932Z 2022-03-17T19:42:12.4611573Z ---------------------------------------------------------------------- 2022-03-17T19:42:12.4612165Z Ran 1 test in 8.417s 2022-03-17T19:42:12.4612471Z 2022-03-17T19:42:12.4612632Z OK 2022-03-17T19:42:12.4612875Z 2022-03-17T19:42:12.4613104Z Generating XML reports... 2022-03-17T19:42:12.4657572Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194204.xml 2022-03-17T19:42:13.5040845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqew2ttl 2022-03-17T19:42:13.5042286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqew2ttl/_remote_module_non_sriptable.py 2022-03-17T19:42:13.8563849Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:42:13.8579428Z 2022-03-17T19:42:13.8579856Z Running tests... 2022-03-17T19:42:13.8580582Z ---------------------------------------------------------------------- 2022-03-17T19:42:15.3651673Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:42:15.4071136Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59060 2022-03-17T19:42:15.4192965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59061 2022-03-17T19:42:15.4315993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59062 2022-03-17T19:42:15.4440513Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59063 2022-03-17T19:42:16.2407293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp3i6cbi 2022-03-17T19:42:16.2409188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp3i6cbi/_remote_module_non_sriptable.py 2022-03-17T19:42:16.2805054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pidoyzb 2022-03-17T19:42:16.2807358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pidoyzb/_remote_module_non_sriptable.py 2022-03-17T19:42:16.2830250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexlqtey1 2022-03-17T19:42:16.2833208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexlqtey1/_remote_module_non_sriptable.py 2022-03-17T19:42:16.3172508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9uyunvuw 2022-03-17T19:42:16.3175051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9uyunvuw/_remote_module_non_sriptable.py 2022-03-17T19:42:16.5818618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:16.6285367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:16.6297762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:16.6656614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:22.3595934Z ok (8.501s) 2022-03-17T19:42:22.3596237Z 2022-03-17T19:42:22.3596648Z ---------------------------------------------------------------------- 2022-03-17T19:42:22.3596976Z Ran 1 test in 8.502s 2022-03-17T19:42:22.3597089Z 2022-03-17T19:42:22.3597179Z OK 2022-03-17T19:42:22.3597312Z 2022-03-17T19:42:22.3597458Z Generating XML reports... 2022-03-17T19:42:22.3641425Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194213.xml 2022-03-17T19:42:23.4044119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9it8_qv7 2022-03-17T19:42:23.4045422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9it8_qv7/_remote_module_non_sriptable.py 2022-03-17T19:42:23.7620700Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:42:23.7626225Z 2022-03-17T19:42:23.7626391Z Running tests... 2022-03-17T19:42:23.7626868Z ---------------------------------------------------------------------- 2022-03-17T19:42:25.3016822Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:42:25.3435399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59417 2022-03-17T19:42:25.3555615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59418 2022-03-17T19:42:25.3678959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59419 2022-03-17T19:42:25.3807102Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59420 2022-03-17T19:42:26.1839693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9wpqrw9 2022-03-17T19:42:26.1840711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9wpqrw9/_remote_module_non_sriptable.py 2022-03-17T19:42:26.2305767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1nfjb0b 2022-03-17T19:42:26.2307975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1nfjb0b/_remote_module_non_sriptable.py 2022-03-17T19:42:26.2620253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchy14ruq 2022-03-17T19:42:26.2623767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchy14ruq/_remote_module_non_sriptable.py 2022-03-17T19:42:26.2638600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposauesi1 2022-03-17T19:42:26.2641419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposauesi1/_remote_module_non_sriptable.py 2022-03-17T19:42:26.5347168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:26.5790590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:26.6091555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:26.6119337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:32.2966943Z ok (8.534s) 2022-03-17T19:42:32.2967178Z 2022-03-17T19:42:32.2967606Z ---------------------------------------------------------------------- 2022-03-17T19:42:32.2967922Z Ran 1 test in 8.534s 2022-03-17T19:42:32.2968107Z 2022-03-17T19:42:32.2968179Z OK 2022-03-17T19:42:32.2968299Z 2022-03-17T19:42:32.2968425Z Generating XML reports... 2022-03-17T19:42:32.3012069Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194223.xml 2022-03-17T19:42:33.3444224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9i95r72 2022-03-17T19:42:33.3445572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9i95r72/_remote_module_non_sriptable.py 2022-03-17T19:42:33.7001602Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:42:33.7017090Z 2022-03-17T19:42:33.7017558Z Running tests... 2022-03-17T19:42:33.7018052Z ---------------------------------------------------------------------- 2022-03-17T19:42:35.2475026Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:42:35.2895302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59774 2022-03-17T19:42:35.3017271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59775 2022-03-17T19:42:35.3143226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59776 2022-03-17T19:42:35.3267127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59777 2022-03-17T19:42:36.1242113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0tjd4fc1 2022-03-17T19:42:36.1243497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0tjd4fc1/_remote_module_non_sriptable.py 2022-03-17T19:42:36.1639786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpib3bir0p 2022-03-17T19:42:36.1642429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpib3bir0p/_remote_module_non_sriptable.py 2022-03-17T19:42:36.1697357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tcfho56 2022-03-17T19:42:36.1700217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tcfho56/_remote_module_non_sriptable.py 2022-03-17T19:42:36.1924025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4sdjx_rg 2022-03-17T19:42:36.1926368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4sdjx_rg/_remote_module_non_sriptable.py 2022-03-17T19:42:36.4871112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:36.5063856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:36.5152391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:36.5403531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:42.2440088Z ok (8.542s) 2022-03-17T19:42:42.2440292Z 2022-03-17T19:42:42.2440731Z ---------------------------------------------------------------------- 2022-03-17T19:42:42.2441370Z Ran 1 test in 8.542s 2022-03-17T19:42:42.2441537Z 2022-03-17T19:42:42.2441609Z OK 2022-03-17T19:42:42.2441746Z 2022-03-17T19:42:42.2441881Z Generating XML reports... 2022-03-17T19:42:42.2485453Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194233.xml 2022-03-17T19:42:43.2835795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28lpuh58 2022-03-17T19:42:43.2837680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28lpuh58/_remote_module_non_sriptable.py 2022-03-17T19:42:43.6403230Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:42:43.6418864Z 2022-03-17T19:42:43.6419166Z Running tests... 2022-03-17T19:42:43.6419646Z ---------------------------------------------------------------------- 2022-03-17T19:42:45.1781839Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:42:45.2201800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60131 2022-03-17T19:42:45.2321522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60132 2022-03-17T19:42:45.2447252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60133 2022-03-17T19:42:45.2575871Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60134 2022-03-17T19:42:46.0323438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26wouerd 2022-03-17T19:42:46.0324333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26wouerd/_remote_module_non_sriptable.py 2022-03-17T19:42:46.0603532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xu8wuat 2022-03-17T19:42:46.0606115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xu8wuat/_remote_module_non_sriptable.py 2022-03-17T19:42:46.0840027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tm3_489 2022-03-17T19:42:46.0843289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tm3_489/_remote_module_non_sriptable.py 2022-03-17T19:42:46.0998685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jq8198g 2022-03-17T19:42:46.1001769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jq8198g/_remote_module_non_sriptable.py 2022-03-17T19:42:46.3822045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:46.4081378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:46.4297482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:42:46.4527458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:52.1748810Z ok (8.533s) 2022-03-17T19:42:52.1749043Z 2022-03-17T19:42:52.1749727Z ---------------------------------------------------------------------- 2022-03-17T19:42:52.1750122Z Ran 1 test in 8.533s 2022-03-17T19:42:52.1750274Z 2022-03-17T19:42:52.1750313Z OK 2022-03-17T19:42:52.1750452Z 2022-03-17T19:42:52.1750570Z Generating XML reports... 2022-03-17T19:42:52.1793365Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194243.xml 2022-03-17T19:42:53.2305519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptv1891eo 2022-03-17T19:42:53.2306190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptv1891eo/_remote_module_non_sriptable.py 2022-03-17T19:42:53.5849349Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:42:53.5864131Z 2022-03-17T19:42:53.5864488Z Running tests... 2022-03-17T19:42:55.1331236Z ---------------------------------------------------------------------- 2022-03-17T19:42:55.1331828Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:42:55.1759796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60488 2022-03-17T19:42:55.1883594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60489 2022-03-17T19:42:55.2011814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60490 2022-03-17T19:42:55.2137188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60491 2022-03-17T19:42:55.9966148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06ofwjka 2022-03-17T19:42:55.9967319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06ofwjka/_remote_module_non_sriptable.py 2022-03-17T19:42:56.0193877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4zlxpg4 2022-03-17T19:42:56.0194944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34vlc46f 2022-03-17T19:42:56.0196241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4zlxpg4/_remote_module_non_sriptable.py 2022-03-17T19:42:56.0197718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34vlc46f/_remote_module_non_sriptable.py 2022-03-17T19:42:56.0803437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpys3nwuzo 2022-03-17T19:42:56.0805336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpys3nwuzo/_remote_module_non_sriptable.py 2022-03-17T19:42:56.3492276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:42:56.3625175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:42:56.3655226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:42:56.4300243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:02.1293904Z ok (8.543s) 2022-03-17T19:43:02.1294231Z 2022-03-17T19:43:02.1294802Z ---------------------------------------------------------------------- 2022-03-17T19:43:02.1295125Z Ran 1 test in 8.543s 2022-03-17T19:43:02.1295299Z 2022-03-17T19:43:02.1295401Z OK 2022-03-17T19:43:02.1295552Z 2022-03-17T19:43:02.1295688Z Generating XML reports... 2022-03-17T19:43:02.1339883Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194253.xml 2022-03-17T19:43:03.1797507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphci0yixp 2022-03-17T19:43:03.1798405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphci0yixp/_remote_module_non_sriptable.py 2022-03-17T19:43:03.5400122Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:43:03.5415568Z 2022-03-17T19:43:03.5415802Z Running tests... 2022-03-17T19:43:03.5416226Z ---------------------------------------------------------------------- 2022-03-17T19:43:05.0662314Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:43:05.1081405Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60845 2022-03-17T19:43:05.1203078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60846 2022-03-17T19:43:05.1326306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60847 2022-03-17T19:43:05.1451848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60848 2022-03-17T19:43:05.9224561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw0ev_6ws 2022-03-17T19:43:05.9225969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw0ev_6ws/_remote_module_non_sriptable.py 2022-03-17T19:43:05.9261923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gfrddqu 2022-03-17T19:43:05.9265234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gfrddqu/_remote_module_non_sriptable.py 2022-03-17T19:43:05.9654540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6k4ailmm 2022-03-17T19:43:05.9656786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6k4ailmm/_remote_module_non_sriptable.py 2022-03-17T19:43:05.9812951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbvpe9g7 2022-03-17T19:43:05.9816093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbvpe9g7/_remote_module_non_sriptable.py 2022-03-17T19:43:06.2751220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:06.2785139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:06.3103350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:06.3242885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:11.9638206Z ok (8.422s) 2022-03-17T19:43:11.9638434Z 2022-03-17T19:43:11.9638805Z ---------------------------------------------------------------------- 2022-03-17T19:43:11.9639142Z Ran 1 test in 8.422s 2022-03-17T19:43:11.9639310Z 2022-03-17T19:43:11.9639402Z OK 2022-03-17T19:43:11.9639518Z 2022-03-17T19:43:11.9639653Z Generating XML reports... 2022-03-17T19:43:11.9682752Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194303.xml 2022-03-17T19:43:13.0094992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizejcby1 2022-03-17T19:43:13.0095841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizejcby1/_remote_module_non_sriptable.py 2022-03-17T19:43:13.3634751Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:43:13.3649950Z 2022-03-17T19:43:13.3650405Z Running tests... 2022-03-17T19:43:13.3650914Z ---------------------------------------------------------------------- 2022-03-17T19:43:14.8976723Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:43:14.9394332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61202 2022-03-17T19:43:14.9515966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61203 2022-03-17T19:43:14.9638842Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61204 2022-03-17T19:43:14.9767730Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61205 2022-03-17T19:43:15.7510576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc44pu9u7 2022-03-17T19:43:15.7511507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc44pu9u7/_remote_module_non_sriptable.py 2022-03-17T19:43:15.7924480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbquij5y1 2022-03-17T19:43:15.7927362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbquij5y1/_remote_module_non_sriptable.py 2022-03-17T19:43:15.8052437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp014riwc3 2022-03-17T19:43:15.8055066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp014riwc3/_remote_module_non_sriptable.py 2022-03-17T19:43:15.8091113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfxm7527x 2022-03-17T19:43:15.8094457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfxm7527x/_remote_module_non_sriptable.py 2022-03-17T19:43:16.1000679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:16.1397845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:16.1511831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:16.1585747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:19.4883986Z ok (6.123s) 2022-03-17T19:43:19.4884485Z 2022-03-17T19:43:19.4885162Z ---------------------------------------------------------------------- 2022-03-17T19:43:19.4885546Z Ran 1 test in 6.123s 2022-03-17T19:43:19.4885679Z 2022-03-17T19:43:19.4885772Z OK 2022-03-17T19:43:19.4885906Z 2022-03-17T19:43:19.4886020Z Generating XML reports... 2022-03-17T19:43:19.4930241Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194313.xml 2022-03-17T19:43:20.4858814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcsastep 2022-03-17T19:43:20.4859973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcsastep/_remote_module_non_sriptable.py 2022-03-17T19:43:20.8443764Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:43:20.8458931Z 2022-03-17T19:43:20.8459374Z Running tests... 2022-03-17T19:43:20.8459860Z ---------------------------------------------------------------------- 2022-03-17T19:43:22.4063403Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:43:22.4484084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61555 2022-03-17T19:43:22.4607966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61556 2022-03-17T19:43:22.4730865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61557 2022-03-17T19:43:22.4856997Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61558 2022-03-17T19:43:23.3008713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ajgv7i5 2022-03-17T19:43:23.3009783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ajgv7i5/_remote_module_non_sriptable.py 2022-03-17T19:43:23.3320546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zc9i_qk 2022-03-17T19:43:23.3323111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zc9i_qk/_remote_module_non_sriptable.py 2022-03-17T19:43:23.3370997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8lo4x70 2022-03-17T19:43:23.3373500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8lo4x70/_remote_module_non_sriptable.py 2022-03-17T19:43:23.3433283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybe4fhol 2022-03-17T19:43:23.3436066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybe4fhol/_remote_module_non_sriptable.py 2022-03-17T19:43:23.6477258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:23.6839809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:23.6843025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:23.6881852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:29.4012102Z ok (8.555s) 2022-03-17T19:43:29.4012308Z 2022-03-17T19:43:29.4012689Z ---------------------------------------------------------------------- 2022-03-17T19:43:29.4013013Z Ran 1 test in 8.555s 2022-03-17T19:43:29.4013190Z 2022-03-17T19:43:29.4013298Z OK 2022-03-17T19:43:29.4013441Z 2022-03-17T19:43:29.4013556Z Generating XML reports... 2022-03-17T19:43:29.4058205Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194320.xml 2022-03-17T19:43:30.4406330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0jbt4gq 2022-03-17T19:43:30.4407670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0jbt4gq/_remote_module_non_sriptable.py 2022-03-17T19:43:30.7957593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:43:30.7973132Z 2022-03-17T19:43:30.7973408Z Running tests... 2022-03-17T19:43:30.7973881Z ---------------------------------------------------------------------- 2022-03-17T19:43:32.3124600Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:43:32.3542506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61920 2022-03-17T19:43:32.3663069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61921 2022-03-17T19:43:32.3787008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61922 2022-03-17T19:43:32.3910855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61923 2022-03-17T19:43:33.1787898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztyi8kh9 2022-03-17T19:43:33.1788939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztyi8kh9/_remote_module_non_sriptable.py 2022-03-17T19:43:33.1816858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqxvp6qa 2022-03-17T19:43:33.1817372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgi8yoqx 2022-03-17T19:43:33.1820018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgi8yoqx/_remote_module_non_sriptable.py 2022-03-17T19:43:33.1820627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqxvp6qa/_remote_module_non_sriptable.py 2022-03-17T19:43:33.2078247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphm8mr921 2022-03-17T19:43:33.2080937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphm8mr921/_remote_module_non_sriptable.py 2022-03-17T19:43:33.5254326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:33.5291869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:33.5359379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:33.5520310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:37.0029530Z ok (6.205s) 2022-03-17T19:43:37.0029760Z 2022-03-17T19:43:37.0030222Z ---------------------------------------------------------------------- 2022-03-17T19:43:37.0030710Z Ran 1 test in 6.206s 2022-03-17T19:43:37.0030994Z 2022-03-17T19:43:37.0031150Z OK 2022-03-17T19:43:37.0035213Z 2022-03-17T19:43:37.0035602Z Generating XML reports... 2022-03-17T19:43:37.0080766Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194330.xml 2022-03-17T19:43:38.0485224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0og3ezz 2022-03-17T19:43:38.0486547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0og3ezz/_remote_module_non_sriptable.py 2022-03-17T19:43:38.4042829Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:43:38.4058582Z 2022-03-17T19:43:38.4058920Z Running tests... 2022-03-17T19:43:38.4059376Z ---------------------------------------------------------------------- 2022-03-17T19:43:39.9393778Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:43:39.9812133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62277 2022-03-17T19:43:39.9932804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62278 2022-03-17T19:43:40.0057123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62279 2022-03-17T19:43:40.0187049Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62280 2022-03-17T19:43:40.9135241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zf20c60 2022-03-17T19:43:40.9135823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwir562eg 2022-03-17T19:43:40.9136342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zf20c60/_remote_module_non_sriptable.py 2022-03-17T19:43:40.9137487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwir562eg/_remote_module_non_sriptable.py 2022-03-17T19:43:40.9279240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e29tvfs 2022-03-17T19:43:40.9281919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e29tvfs/_remote_module_non_sriptable.py 2022-03-17T19:43:40.9400143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbnqnc_w 2022-03-17T19:43:40.9402784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbnqnc_w/_remote_module_non_sriptable.py 2022-03-17T19:43:41.2556445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:41.2611178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:41.2770867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:41.2899287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:44.7298080Z ok (6.324s) 2022-03-17T19:43:44.7298444Z 2022-03-17T19:43:44.7298827Z ---------------------------------------------------------------------- 2022-03-17T19:43:44.7299122Z Ran 1 test in 6.324s 2022-03-17T19:43:44.7299290Z 2022-03-17T19:43:44.7299411Z OK 2022-03-17T19:43:44.7299528Z 2022-03-17T19:43:44.7299654Z Generating XML reports... 2022-03-17T19:43:44.7344821Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194338.xml 2022-03-17T19:43:45.7801313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69noaexj 2022-03-17T19:43:45.7802391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69noaexj/_remote_module_non_sriptable.py 2022-03-17T19:43:46.1376154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:43:46.1393841Z 2022-03-17T19:43:46.1394258Z Running tests... 2022-03-17T19:43:46.1394781Z ---------------------------------------------------------------------- 2022-03-17T19:43:47.6667251Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:43:47.7087993Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62634 2022-03-17T19:43:47.7208888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62635 2022-03-17T19:43:47.7332949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62636 2022-03-17T19:43:47.7457935Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62637 2022-03-17T19:43:48.5204036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47dj6sii 2022-03-17T19:43:48.5205298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47dj6sii/_remote_module_non_sriptable.py 2022-03-17T19:43:48.5327755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkxdb86c 2022-03-17T19:43:48.5330743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkxdb86c/_remote_module_non_sriptable.py 2022-03-17T19:43:48.5622015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdiimssf 2022-03-17T19:43:48.5624675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdiimssf/_remote_module_non_sriptable.py 2022-03-17T19:43:48.5782754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcz8t0zlv 2022-03-17T19:43:48.5785544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcz8t0zlv/_remote_module_non_sriptable.py 2022-03-17T19:43:48.8678296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:48.8843499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:43:48.9058876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:48.9252578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:54.6609167Z ok (8.521s) 2022-03-17T19:43:54.6609364Z 2022-03-17T19:43:54.6609798Z ---------------------------------------------------------------------- 2022-03-17T19:43:54.6610146Z Ran 1 test in 8.522s 2022-03-17T19:43:54.6610342Z 2022-03-17T19:43:54.6610436Z OK 2022-03-17T19:43:54.6610542Z 2022-03-17T19:43:54.6610697Z Generating XML reports... 2022-03-17T19:43:54.6657927Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194346.xml 2022-03-17T19:43:55.7079512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25e5rnme 2022-03-17T19:43:55.7080699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25e5rnme/_remote_module_non_sriptable.py 2022-03-17T19:43:56.0661345Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:43:56.0677132Z 2022-03-17T19:43:56.0677310Z Running tests... 2022-03-17T19:43:56.0677774Z ---------------------------------------------------------------------- 2022-03-17T19:43:57.6233378Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:43:57.6653035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62999 2022-03-17T19:43:57.6775534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63000 2022-03-17T19:43:57.6897240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63001 2022-03-17T19:43:57.7021441Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63002 2022-03-17T19:43:58.5378535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppi2_txhw 2022-03-17T19:43:58.5379120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppi2_txhw/_remote_module_non_sriptable.py 2022-03-17T19:43:58.5382501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8692pn5g 2022-03-17T19:43:58.5385814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8692pn5g/_remote_module_non_sriptable.py 2022-03-17T19:43:58.5645781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_n454su 2022-03-17T19:43:58.5648387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_n454su/_remote_module_non_sriptable.py 2022-03-17T19:43:58.5784601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiul1gai 2022-03-17T19:43:58.5787160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiul1gai/_remote_module_non_sriptable.py 2022-03-17T19:43:58.8846564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:43:58.8880539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:43:58.9095952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:43:58.9273114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:04.6176987Z ok (8.550s) 2022-03-17T19:44:04.6177265Z 2022-03-17T19:44:04.6177682Z ---------------------------------------------------------------------- 2022-03-17T19:44:04.6178413Z Ran 1 test in 8.550s 2022-03-17T19:44:04.6178563Z 2022-03-17T19:44:04.6178667Z OK 2022-03-17T19:44:04.6178811Z 2022-03-17T19:44:04.6178957Z Generating XML reports... 2022-03-17T19:44:04.6222765Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194356.xml 2022-03-17T19:44:05.6546148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2101goi9 2022-03-17T19:44:05.6547041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2101goi9/_remote_module_non_sriptable.py 2022-03-17T19:44:06.0065588Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:06.0080355Z 2022-03-17T19:44:06.0080604Z Running tests... 2022-03-17T19:44:06.0081025Z ---------------------------------------------------------------------- 2022-03-17T19:44:07.5114012Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:07.5528108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63364 2022-03-17T19:44:07.5647927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63365 2022-03-17T19:44:07.5770086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63366 2022-03-17T19:44:07.5894259Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63367 2022-03-17T19:44:08.3714983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcm7dhvh 2022-03-17T19:44:08.3716285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcm7dhvh/_remote_module_non_sriptable.py 2022-03-17T19:44:08.3728585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfesp5u3 2022-03-17T19:44:08.3731087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfesp5u3/_remote_module_non_sriptable.py 2022-03-17T19:44:08.3789530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2m3romc 2022-03-17T19:44:08.3792462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2m3romc/_remote_module_non_sriptable.py 2022-03-17T19:44:08.4619208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmavoc92a 2022-03-17T19:44:08.4621434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmavoc92a/_remote_module_non_sriptable.py 2022-03-17T19:44:08.7153648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:08.7188540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:08.7202768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:08.8122483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:09.0940860Z ok (3.086s) 2022-03-17T19:44:09.0941114Z 2022-03-17T19:44:09.0941761Z ---------------------------------------------------------------------- 2022-03-17T19:44:09.0942087Z Ran 1 test in 3.086s 2022-03-17T19:44:09.0942246Z 2022-03-17T19:44:09.0942327Z OK 2022-03-17T19:44:09.0942462Z 2022-03-17T19:44:09.0942592Z Generating XML reports... 2022-03-17T19:44:09.0984944Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194406.xml 2022-03-17T19:44:10.1351353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5nqxxmo 2022-03-17T19:44:10.1352866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5nqxxmo/_remote_module_non_sriptable.py 2022-03-17T19:44:10.4879208Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:10.4894122Z 2022-03-17T19:44:10.4894321Z Running tests... 2022-03-17T19:44:10.4894959Z ---------------------------------------------------------------------- 2022-03-17T19:44:12.0335057Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:12.0754187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63557 2022-03-17T19:44:12.0875186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63558 2022-03-17T19:44:12.1002360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63559 2022-03-17T19:44:12.1127439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63560 2022-03-17T19:44:12.8924878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70aat9rw 2022-03-17T19:44:12.8926109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70aat9rw/_remote_module_non_sriptable.py 2022-03-17T19:44:12.9070132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc4f_3ro 2022-03-17T19:44:12.9072803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc4f_3ro/_remote_module_non_sriptable.py 2022-03-17T19:44:12.9350165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpig_58xzy 2022-03-17T19:44:12.9352696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpig_58xzy/_remote_module_non_sriptable.py 2022-03-17T19:44:12.9513752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbaamhotn 2022-03-17T19:44:12.9516354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbaamhotn/_remote_module_non_sriptable.py 2022-03-17T19:44:13.2414657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:13.2574865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:13.2852515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:13.2958198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:13.5173450Z ok (3.028s) 2022-03-17T19:44:13.5173691Z 2022-03-17T19:44:13.5174285Z ---------------------------------------------------------------------- 2022-03-17T19:44:13.5174791Z Ran 1 test in 3.028s 2022-03-17T19:44:13.5175108Z 2022-03-17T19:44:13.5175222Z OK 2022-03-17T19:44:13.5175342Z 2022-03-17T19:44:13.5175481Z Generating XML reports... 2022-03-17T19:44:13.5218607Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194410.xml 2022-03-17T19:44:14.5684248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjg2v654w 2022-03-17T19:44:14.5685379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjg2v654w/_remote_module_non_sriptable.py 2022-03-17T19:44:14.9220711Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:14.9236006Z 2022-03-17T19:44:14.9236483Z Running tests... 2022-03-17T19:44:14.9236981Z ---------------------------------------------------------------------- 2022-03-17T19:44:16.4581344Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:16.5001755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63750 2022-03-17T19:44:16.5123223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63751 2022-03-17T19:44:16.5248635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63752 2022-03-17T19:44:16.5377306Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63753 2022-03-17T19:44:17.3088143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmw88dzb1 2022-03-17T19:44:17.3089240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmw88dzb1/_remote_module_non_sriptable.py 2022-03-17T19:44:17.3479888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpun_41upy 2022-03-17T19:44:17.3482028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpun_41upy/_remote_module_non_sriptable.py 2022-03-17T19:44:17.3546527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9wgij5b 2022-03-17T19:44:17.3549391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9wgij5b/_remote_module_non_sriptable.py 2022-03-17T19:44:17.3821336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn91ihvra 2022-03-17T19:44:17.3824445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn91ihvra/_remote_module_non_sriptable.py 2022-03-17T19:44:17.6587721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:17.6995019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:17.6995533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:17.7332314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:17.9423300Z ok (3.018s) 2022-03-17T19:44:17.9423534Z 2022-03-17T19:44:17.9423916Z ---------------------------------------------------------------------- 2022-03-17T19:44:17.9424254Z Ran 1 test in 3.019s 2022-03-17T19:44:17.9424415Z 2022-03-17T19:44:17.9424515Z OK 2022-03-17T19:44:17.9424632Z 2022-03-17T19:44:17.9424766Z Generating XML reports... 2022-03-17T19:44:17.9467861Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194414.xml 2022-03-17T19:44:18.9934641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnraiukug 2022-03-17T19:44:18.9935732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnraiukug/_remote_module_non_sriptable.py 2022-03-17T19:44:19.3522040Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:19.3538742Z 2022-03-17T19:44:19.3539192Z Running tests... 2022-03-17T19:44:19.3539679Z ---------------------------------------------------------------------- 2022-03-17T19:44:20.8867506Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:20.9291737Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63931 2022-03-17T19:44:20.9411851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63932 2022-03-17T19:44:20.9535410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63933 2022-03-17T19:44:20.9664116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63934 2022-03-17T19:44:21.8086631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5y16y1v 2022-03-17T19:44:21.8087596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5y16y1v/_remote_module_non_sriptable.py 2022-03-17T19:44:21.8097699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_is66nm5 2022-03-17T19:44:21.8100517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_is66nm5/_remote_module_non_sriptable.py 2022-03-17T19:44:21.8122894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppuv06gra 2022-03-17T19:44:21.8126143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppuv06gra/_remote_module_non_sriptable.py 2022-03-17T19:44:21.8236608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx74azsj7 2022-03-17T19:44:21.8239021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx74azsj7/_remote_module_non_sriptable.py 2022-03-17T19:44:22.1574102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:22.1615950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:22.1704547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:22.1725587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:22.4714228Z ok (3.117s) 2022-03-17T19:44:22.4714561Z 2022-03-17T19:44:22.4715315Z ---------------------------------------------------------------------- 2022-03-17T19:44:22.4716039Z Ran 1 test in 3.118s 2022-03-17T19:44:22.4716205Z 2022-03-17T19:44:22.4716303Z OK 2022-03-17T19:44:22.4716782Z 2022-03-17T19:44:22.4718767Z Generating XML reports... 2022-03-17T19:44:22.4761165Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194419.xml 2022-03-17T19:44:23.5274663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkolzpueg 2022-03-17T19:44:23.5275737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkolzpueg/_remote_module_non_sriptable.py 2022-03-17T19:44:23.8904125Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:23.8919720Z 2022-03-17T19:44:23.8920173Z Running tests... 2022-03-17T19:44:23.8920689Z ---------------------------------------------------------------------- 2022-03-17T19:44:25.4418051Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:25.4840714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64124 2022-03-17T19:44:25.4961071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64125 2022-03-17T19:44:25.5087965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64126 2022-03-17T19:44:25.5212748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64127 2022-03-17T19:44:26.3574989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7xyxg6n 2022-03-17T19:44:26.3575862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7xyxg6n/_remote_module_non_sriptable.py 2022-03-17T19:44:26.3654279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgro1jop6 2022-03-17T19:44:26.3656843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgro1jop6/_remote_module_non_sriptable.py 2022-03-17T19:44:26.3747133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8i__qk6 2022-03-17T19:44:26.3749582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8i__qk6/_remote_module_non_sriptable.py 2022-03-17T19:44:26.3777038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplk2c5fhl 2022-03-17T19:44:26.3780034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplk2c5fhl/_remote_module_non_sriptable.py 2022-03-17T19:44:26.7011314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:26.7091719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:26.7224993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:26.7263554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:29.0298883Z ok (5.138s) 2022-03-17T19:44:29.0299122Z 2022-03-17T19:44:29.0299508Z ---------------------------------------------------------------------- 2022-03-17T19:44:29.0299828Z Ran 1 test in 5.138s 2022-03-17T19:44:29.0299994Z 2022-03-17T19:44:29.0300087Z OK 2022-03-17T19:44:29.0300504Z 2022-03-17T19:44:29.0300652Z Generating XML reports... 2022-03-17T19:44:29.0344389Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194423.xml 2022-03-17T19:44:30.0891996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk1e9q7w 2022-03-17T19:44:30.0893172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk1e9q7w/_remote_module_non_sriptable.py 2022-03-17T19:44:30.4456369Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:30.4471975Z 2022-03-17T19:44:30.4472256Z Running tests... 2022-03-17T19:44:30.4472675Z ---------------------------------------------------------------------- 2022-03-17T19:44:32.0018890Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:32.0450659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64477 2022-03-17T19:44:32.0575158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64478 2022-03-17T19:44:32.0701273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64479 2022-03-17T19:44:32.0830230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64480 2022-03-17T19:44:32.8629011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl2txyk4 2022-03-17T19:44:32.8629643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl2txyk4/_remote_module_non_sriptable.py 2022-03-17T19:44:32.8945002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln181725 2022-03-17T19:44:32.8947535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln181725/_remote_module_non_sriptable.py 2022-03-17T19:44:32.9131971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoirmc6eo 2022-03-17T19:44:32.9135269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoirmc6eo/_remote_module_non_sriptable.py 2022-03-17T19:44:32.9178642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zguz0vh 2022-03-17T19:44:32.9181724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zguz0vh/_remote_module_non_sriptable.py 2022-03-17T19:44:33.2139236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:33.2400085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:33.2531091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:33.2692887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:35.6919077Z ok (5.244s) 2022-03-17T19:44:35.6919316Z 2022-03-17T19:44:35.6919906Z ---------------------------------------------------------------------- 2022-03-17T19:44:35.6920315Z Ran 1 test in 5.245s 2022-03-17T19:44:35.6920486Z 2022-03-17T19:44:35.6920568Z OK 2022-03-17T19:44:35.6920705Z 2022-03-17T19:44:35.6920841Z Generating XML reports... 2022-03-17T19:44:35.6963695Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194430.xml 2022-03-17T19:44:36.7472189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzybhz4f7 2022-03-17T19:44:36.7473069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzybhz4f7/_remote_module_non_sriptable.py 2022-03-17T19:44:37.1075311Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:37.1090400Z 2022-03-17T19:44:37.1090762Z Running tests... 2022-03-17T19:44:37.1091247Z ---------------------------------------------------------------------- 2022-03-17T19:44:38.6414328Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:38.6843889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64830 2022-03-17T19:44:38.6964763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64831 2022-03-17T19:44:38.7087746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64832 2022-03-17T19:44:38.7217703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64833 2022-03-17T19:44:39.5500125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3ncg48h 2022-03-17T19:44:39.5501198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3ncg48h/_remote_module_non_sriptable.py 2022-03-17T19:44:39.5522900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hqat_im 2022-03-17T19:44:39.5526307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hqat_im/_remote_module_non_sriptable.py 2022-03-17T19:44:39.5792616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi96qz6lg 2022-03-17T19:44:39.5795629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi96qz6lg/_remote_module_non_sriptable.py 2022-03-17T19:44:39.5800635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jqwopdh 2022-03-17T19:44:39.5803924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jqwopdh/_remote_module_non_sriptable.py 2022-03-17T19:44:39.8954565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:39.9003576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:39.9237738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:39.9286001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:42.2304391Z ok (5.121s) 2022-03-17T19:44:42.2304595Z 2022-03-17T19:44:42.2305007Z ---------------------------------------------------------------------- 2022-03-17T19:44:42.2305329Z Ran 1 test in 5.121s 2022-03-17T19:44:42.2305514Z 2022-03-17T19:44:42.2305601Z OK 2022-03-17T19:44:42.2305742Z 2022-03-17T19:44:42.2305874Z Generating XML reports... 2022-03-17T19:44:42.2349176Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194437.xml 2022-03-17T19:44:43.2748278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_q498ht4 2022-03-17T19:44:43.2749236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_q498ht4/_remote_module_non_sriptable.py 2022-03-17T19:44:43.6251016Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:43.6267233Z 2022-03-17T19:44:43.6267590Z Running tests... 2022-03-17T19:44:43.6268057Z ---------------------------------------------------------------------- 2022-03-17T19:44:45.1570164Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:45.2001625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65183 2022-03-17T19:44:45.2123619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65184 2022-03-17T19:44:45.2251345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65185 2022-03-17T19:44:45.2381301Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65186 2022-03-17T19:44:46.0417219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps77putzr 2022-03-17T19:44:46.0418280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps77putzr/_remote_module_non_sriptable.py 2022-03-17T19:44:46.0661068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawe4vpja 2022-03-17T19:44:46.0664729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawe4vpja/_remote_module_non_sriptable.py 2022-03-17T19:44:46.0929639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr3hxebx 2022-03-17T19:44:46.0931972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr3hxebx/_remote_module_non_sriptable.py 2022-03-17T19:44:46.1022201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7j9_5fe 2022-03-17T19:44:46.1025366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7j9_5fe/_remote_module_non_sriptable.py 2022-03-17T19:44:46.3872659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:46.4168710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:46.4435428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:46.4504988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:48.6468129Z ok (5.020s) 2022-03-17T19:44:48.6468312Z 2022-03-17T19:44:48.6469020Z ---------------------------------------------------------------------- 2022-03-17T19:44:48.6469316Z Ran 1 test in 5.020s 2022-03-17T19:44:48.6469497Z 2022-03-17T19:44:48.6469632Z OK 2022-03-17T19:44:48.6469726Z 2022-03-17T19:44:48.6469864Z Generating XML reports... 2022-03-17T19:44:48.6512015Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194443.xml 2022-03-17T19:44:49.6986220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpwis_7b 2022-03-17T19:44:49.6987761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpwis_7b/_remote_module_non_sriptable.py 2022-03-17T19:44:50.0534762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:50.0551068Z 2022-03-17T19:44:50.0551536Z Running tests... 2022-03-17T19:44:50.0552026Z ---------------------------------------------------------------------- 2022-03-17T19:44:51.5998173Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:51.6420055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65536 2022-03-17T19:44:51.6541009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65537 2022-03-17T19:44:51.6666966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65538 2022-03-17T19:44:51.6790700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65539 2022-03-17T19:44:52.5336043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwql0s2i 2022-03-17T19:44:52.5337099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwql0s2i/_remote_module_non_sriptable.py 2022-03-17T19:44:52.5521869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf56135e8 2022-03-17T19:44:52.5524396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf56135e8/_remote_module_non_sriptable.py 2022-03-17T19:44:52.5626649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xofauh4 2022-03-17T19:44:52.5629664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxms9vmm 2022-03-17T19:44:52.5630197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xofauh4/_remote_module_non_sriptable.py 2022-03-17T19:44:52.5632481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxms9vmm/_remote_module_non_sriptable.py 2022-03-17T19:44:52.8788409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:44:52.8951259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:52.9130434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:52.9147592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:55.1882065Z ok (5.133s) 2022-03-17T19:44:55.1882404Z 2022-03-17T19:44:55.1882913Z ---------------------------------------------------------------------- 2022-03-17T19:44:55.1883593Z Ran 1 test in 5.133s 2022-03-17T19:44:55.1883761Z 2022-03-17T19:44:55.1883858Z OK 2022-03-17T19:44:55.1883998Z 2022-03-17T19:44:55.1884111Z Generating XML reports... 2022-03-17T19:44:55.1927990Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194450.xml 2022-03-17T19:44:56.2084262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwii_nqs 2022-03-17T19:44:56.2085613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwii_nqs/_remote_module_non_sriptable.py 2022-03-17T19:44:56.5681887Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:44:56.5697237Z 2022-03-17T19:44:56.5697468Z Running tests... 2022-03-17T19:44:56.5698100Z ---------------------------------------------------------------------- 2022-03-17T19:44:58.1076373Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:44:58.1498634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65889 2022-03-17T19:44:58.1619136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65890 2022-03-17T19:44:58.1745755Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65891 2022-03-17T19:44:58.1875573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65892 2022-03-17T19:44:59.0548402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiaruphj_ 2022-03-17T19:44:59.0549345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiaruphj_/_remote_module_non_sriptable.py 2022-03-17T19:44:59.0662400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yleit3m 2022-03-17T19:44:59.0665557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yleit3m/_remote_module_non_sriptable.py 2022-03-17T19:44:59.0667522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0zs0b5y 2022-03-17T19:44:59.0671104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0zs0b5y/_remote_module_non_sriptable.py 2022-03-17T19:44:59.0955474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuxo6jfp 2022-03-17T19:44:59.0957955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuxo6jfp/_remote_module_non_sriptable.py 2022-03-17T19:44:59.4056593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:44:59.4088112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:44:59.4271153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:44:59.4389329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:01.6964632Z ok (5.126s) 2022-03-17T19:45:01.6964860Z 2022-03-17T19:45:01.6965883Z ---------------------------------------------------------------------- 2022-03-17T19:45:01.6966574Z Ran 1 test in 5.127s 2022-03-17T19:45:01.6966748Z 2022-03-17T19:45:01.6966846Z OK 2022-03-17T19:45:01.6966987Z 2022-03-17T19:45:01.6967128Z Generating XML reports... 2022-03-17T19:45:01.7010734Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194456.xml 2022-03-17T19:45:02.7595122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpouzamaxy 2022-03-17T19:45:02.7596121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpouzamaxy/_remote_module_non_sriptable.py 2022-03-17T19:45:03.1175816Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:03.1191336Z 2022-03-17T19:45:03.1191456Z Running tests... 2022-03-17T19:45:03.1192203Z ---------------------------------------------------------------------- 2022-03-17T19:45:04.6502724Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:04.6919524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66242 2022-03-17T19:45:04.7041999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66243 2022-03-17T19:45:04.7167178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66244 2022-03-17T19:45:04.7294421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66245 2022-03-17T19:45:05.5038761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnh9mdhb 2022-03-17T19:45:05.5039417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnh9mdhb/_remote_module_non_sriptable.py 2022-03-17T19:45:05.5070904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx42icy41 2022-03-17T19:45:05.5073691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx42icy41/_remote_module_non_sriptable.py 2022-03-17T19:45:05.5080058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyk4whhg 2022-03-17T19:45:05.5083090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyk4whhg/_remote_module_non_sriptable.py 2022-03-17T19:45:05.5650419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qn283ps 2022-03-17T19:45:05.5652832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qn283ps/_remote_module_non_sriptable.py 2022-03-17T19:45:05.8491772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:05.8534983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:05.8569900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:05.9148265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:08.1381674Z ok (5.019s) 2022-03-17T19:45:08.1382064Z 2022-03-17T19:45:08.1382613Z ---------------------------------------------------------------------- 2022-03-17T19:45:08.1382959Z Ran 1 test in 5.019s 2022-03-17T19:45:08.1383109Z 2022-03-17T19:45:08.1383203Z OK 2022-03-17T19:45:08.1383340Z 2022-03-17T19:45:08.1383473Z Generating XML reports... 2022-03-17T19:45:08.1426362Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194503.xml 2022-03-17T19:45:09.1915508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qjlm5c5 2022-03-17T19:45:09.1917900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qjlm5c5/_remote_module_non_sriptable.py 2022-03-17T19:45:09.5474762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:09.5489990Z 2022-03-17T19:45:09.5490398Z Running tests... 2022-03-17T19:45:09.5490894Z ---------------------------------------------------------------------- 2022-03-17T19:45:11.0738480Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:11.1157529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66595 2022-03-17T19:45:11.1278516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66596 2022-03-17T19:45:11.1401481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66597 2022-03-17T19:45:11.1526632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66598 2022-03-17T19:45:11.9426747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0tygwmpk 2022-03-17T19:45:11.9428172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0tygwmpk/_remote_module_non_sriptable.py 2022-03-17T19:45:11.9567432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrhogu_z 2022-03-17T19:45:11.9570132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrhogu_z/_remote_module_non_sriptable.py 2022-03-17T19:45:12.0003046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbd51dld 2022-03-17T19:45:12.0005455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbd51dld/_remote_module_non_sriptable.py 2022-03-17T19:45:12.0025763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn45255xn 2022-03-17T19:45:12.0028815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn45255xn/_remote_module_non_sriptable.py 2022-03-17T19:45:12.2913827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:12.2999264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:12.3441117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:12.3521820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:18.0683128Z ok (8.519s) 2022-03-17T19:45:18.0683369Z 2022-03-17T19:45:18.0683757Z ---------------------------------------------------------------------- 2022-03-17T19:45:18.0684027Z Ran 1 test in 8.519s 2022-03-17T19:45:18.0684169Z 2022-03-17T19:45:18.0684230Z OK 2022-03-17T19:45:18.0684368Z 2022-03-17T19:45:18.0684502Z Generating XML reports... 2022-03-17T19:45:18.0727562Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194509.xml 2022-03-17T19:45:19.1056057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurs25rum 2022-03-17T19:45:19.1057250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurs25rum/_remote_module_non_sriptable.py 2022-03-17T19:45:19.4557143Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:19.4572453Z 2022-03-17T19:45:19.4572862Z Running tests... 2022-03-17T19:45:19.4573313Z ---------------------------------------------------------------------- 2022-03-17T19:45:20.9710542Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:21.0131601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66960 2022-03-17T19:45:21.0254833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66961 2022-03-17T19:45:21.0383115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66962 2022-03-17T19:45:21.0513694Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66963 2022-03-17T19:45:21.9397141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqeiirh09 2022-03-17T19:45:21.9398142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqeiirh09/_remote_module_non_sriptable.py 2022-03-17T19:45:21.9399316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxszxxlez 2022-03-17T19:45:21.9402775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxszxxlez/_remote_module_non_sriptable.py 2022-03-17T19:45:21.9408597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwamtih4b 2022-03-17T19:45:21.9411306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4y3jutc9 2022-03-17T19:45:21.9411844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwamtih4b/_remote_module_non_sriptable.py 2022-03-17T19:45:21.9414320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4y3jutc9/_remote_module_non_sriptable.py 2022-03-17T19:45:22.2816468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:22.2849806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:22.2935423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:22.2947738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:28.0672151Z ok (8.610s) 2022-03-17T19:45:28.0672377Z 2022-03-17T19:45:28.0672868Z ---------------------------------------------------------------------- 2022-03-17T19:45:28.0673198Z Ran 1 test in 8.610s 2022-03-17T19:45:28.0673376Z 2022-03-17T19:45:28.0673472Z OK 2022-03-17T19:45:28.0673619Z 2022-03-17T19:45:28.0673765Z Generating XML reports... 2022-03-17T19:45:28.0719192Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194519.xml 2022-03-17T19:45:29.1281403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmtqmn34 2022-03-17T19:45:29.1282808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmtqmn34/_remote_module_non_sriptable.py 2022-03-17T19:45:29.4789954Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:29.4805583Z 2022-03-17T19:45:29.4806020Z Running tests... 2022-03-17T19:45:29.4806500Z ---------------------------------------------------------------------- 2022-03-17T19:45:31.0220509Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:31.0637906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67317 2022-03-17T19:45:31.0761137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67318 2022-03-17T19:45:31.0887548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67319 2022-03-17T19:45:31.1012178Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67320 2022-03-17T19:45:31.9208045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpia9pwx_6 2022-03-17T19:45:31.9208901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpia9pwx_6/_remote_module_non_sriptable.py 2022-03-17T19:45:31.9474194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp095nuqaj 2022-03-17T19:45:31.9476767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp095nuqaj/_remote_module_non_sriptable.py 2022-03-17T19:45:31.9632516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprih1m6ij 2022-03-17T19:45:31.9635905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprih1m6ij/_remote_module_non_sriptable.py 2022-03-17T19:45:31.9729074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiampf57r 2022-03-17T19:45:31.9731863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiampf57r/_remote_module_non_sriptable.py 2022-03-17T19:45:32.2657168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:32.2929208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:32.3081785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:32.3225568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:32.5060197Z ok (3.025s) 2022-03-17T19:45:32.5060865Z 2022-03-17T19:45:32.5061240Z ---------------------------------------------------------------------- 2022-03-17T19:45:32.5061599Z Ran 1 test in 3.025s 2022-03-17T19:45:32.5061895Z 2022-03-17T19:45:32.5062080Z OK 2022-03-17T19:45:32.5062224Z 2022-03-17T19:45:32.5062358Z Generating XML reports... 2022-03-17T19:45:32.5105667Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194529.xml 2022-03-17T19:45:33.5574485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4f0fbn_ 2022-03-17T19:45:33.5575796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4f0fbn_/_remote_module_non_sriptable.py 2022-03-17T19:45:33.9154244Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:33.9169609Z 2022-03-17T19:45:33.9169840Z Running tests... 2022-03-17T19:45:33.9170289Z ---------------------------------------------------------------------- 2022-03-17T19:45:35.4481018Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:35.4896832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67498 2022-03-17T19:45:35.5017828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67499 2022-03-17T19:45:35.5142235Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67500 2022-03-17T19:45:35.5273274Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67501 2022-03-17T19:45:36.3217547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzi6tb8jq 2022-03-17T19:45:36.3218233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzi6tb8jq/_remote_module_non_sriptable.py 2022-03-17T19:45:36.3361659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp344aze7a 2022-03-17T19:45:36.3364472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp344aze7a/_remote_module_non_sriptable.py 2022-03-17T19:45:36.3936915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88jozuic 2022-03-17T19:45:36.3939082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88jozuic/_remote_module_non_sriptable.py 2022-03-17T19:45:36.3956943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08mlptu4 2022-03-17T19:45:36.3960404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08mlptu4/_remote_module_non_sriptable.py 2022-03-17T19:45:36.6726476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:36.6788950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:36.7368870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:36.7510461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:42.4421345Z ok (8.525s) 2022-03-17T19:45:42.4426357Z 2022-03-17T19:45:42.4427136Z ---------------------------------------------------------------------- 2022-03-17T19:45:42.4427685Z Ran 1 test in 8.525s 2022-03-17T19:45:42.4427976Z 2022-03-17T19:45:42.4428135Z OK 2022-03-17T19:45:42.4428354Z 2022-03-17T19:45:42.4428556Z Generating XML reports... 2022-03-17T19:45:42.4470723Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194533.xml 2022-03-17T19:45:43.4901196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gvo8cug 2022-03-17T19:45:43.4902225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gvo8cug/_remote_module_non_sriptable.py 2022-03-17T19:45:43.8453794Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:43.8469844Z 2022-03-17T19:45:43.8470304Z Running tests... 2022-03-17T19:45:43.8470788Z ---------------------------------------------------------------------- 2022-03-17T19:45:45.3798393Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:45.4229027Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67863 2022-03-17T19:45:45.4354129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67864 2022-03-17T19:45:45.4480452Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67865 2022-03-17T19:45:45.4613500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67866 2022-03-17T19:45:46.2890260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbuebbu2 2022-03-17T19:45:46.2890844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbuebbu2/_remote_module_non_sriptable.py 2022-03-17T19:45:46.3489535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6eqgxqej 2022-03-17T19:45:46.3490493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6eqgxqej/_remote_module_non_sriptable.py 2022-03-17T19:45:46.3517505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8xnkhbg 2022-03-17T19:45:46.3520796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8xnkhbg/_remote_module_non_sriptable.py 2022-03-17T19:45:46.3564488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkgn5wno 2022-03-17T19:45:46.3567794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkgn5wno/_remote_module_non_sriptable.py 2022-03-17T19:45:46.6412776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:46.6926466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:46.7034318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:46.7059467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:46.8660774Z skip (3.019s) 2022-03-17T19:45:46.8661487Z 2022-03-17T19:45:46.8661872Z ---------------------------------------------------------------------- 2022-03-17T19:45:46.8662134Z Ran 1 test in 3.019s 2022-03-17T19:45:46.8662300Z 2022-03-17T19:45:46.8662489Z OK (skipped=1) 2022-03-17T19:45:46.8662752Z 2022-03-17T19:45:46.8662964Z Generating XML reports... 2022-03-17T19:45:46.8708535Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194543.xml 2022-03-17T19:45:47.9208258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvptiaek 2022-03-17T19:45:47.9209533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvptiaek/_remote_module_non_sriptable.py 2022-03-17T19:45:48.2721441Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:48.2736619Z 2022-03-17T19:45:48.2737030Z Running tests... 2022-03-17T19:45:48.2737542Z ---------------------------------------------------------------------- 2022-03-17T19:45:49.7983052Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:49.8405038Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68044 2022-03-17T19:45:49.8528033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68045 2022-03-17T19:45:49.8652580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68046 2022-03-17T19:45:49.8778951Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68047 2022-03-17T19:45:50.7164383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9uhhp3r 2022-03-17T19:45:50.7165621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9uhhp3r/_remote_module_non_sriptable.py 2022-03-17T19:45:50.7572342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnvrqyzh 2022-03-17T19:45:50.7575381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnvrqyzh/_remote_module_non_sriptable.py 2022-03-17T19:45:50.7701240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn60ybwzv 2022-03-17T19:45:50.7704116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn60ybwzv/_remote_module_non_sriptable.py 2022-03-17T19:45:50.7724220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9hto0vw 2022-03-17T19:45:50.7727145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9hto0vw/_remote_module_non_sriptable.py 2022-03-17T19:45:51.0714295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:51.1063705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:51.1132962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:51.1228372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:51.2825533Z skip (3.009s) 2022-03-17T19:45:51.2825758Z 2022-03-17T19:45:51.2826208Z ---------------------------------------------------------------------- 2022-03-17T19:45:51.2826483Z Ran 1 test in 3.009s 2022-03-17T19:45:51.2826672Z 2022-03-17T19:45:51.2826787Z OK (skipped=1) 2022-03-17T19:45:51.2826950Z 2022-03-17T19:45:51.2827023Z Generating XML reports... 2022-03-17T19:45:51.2871294Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194548.xml 2022-03-17T19:45:52.3205229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19cxhwdp 2022-03-17T19:45:52.3206817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19cxhwdp/_remote_module_non_sriptable.py 2022-03-17T19:45:52.6770592Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:52.6786316Z 2022-03-17T19:45:52.6786625Z Running tests... 2022-03-17T19:45:52.6787084Z ---------------------------------------------------------------------- 2022-03-17T19:45:54.2046240Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:54.2465060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68225 2022-03-17T19:45:54.2586118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68226 2022-03-17T19:45:54.2711397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68227 2022-03-17T19:45:54.2836748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68228 2022-03-17T19:45:55.0712912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7k895yw 2022-03-17T19:45:55.0713996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwadjmwdr 2022-03-17T19:45:55.0715060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7k895yw/_remote_module_non_sriptable.py 2022-03-17T19:45:55.0716679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwadjmwdr/_remote_module_non_sriptable.py 2022-03-17T19:45:55.1298006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbv6njer3 2022-03-17T19:45:55.1299672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbv6njer3/_remote_module_non_sriptable.py 2022-03-17T19:45:55.1388756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi682o70 2022-03-17T19:45:55.1391593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi682o70/_remote_module_non_sriptable.py 2022-03-17T19:45:55.4203416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:45:55.4233206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:55.4751956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:55.4949899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:55.7886300Z ok (3.110s) 2022-03-17T19:45:55.7886560Z 2022-03-17T19:45:55.7887326Z ---------------------------------------------------------------------- 2022-03-17T19:45:55.7887633Z Ran 1 test in 3.110s 2022-03-17T19:45:55.7887813Z 2022-03-17T19:45:55.7887891Z OK 2022-03-17T19:45:55.7888022Z 2022-03-17T19:45:55.7888141Z Generating XML reports... 2022-03-17T19:45:55.7931289Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194552.xml 2022-03-17T19:45:56.8114318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghetxyc1 2022-03-17T19:45:56.8115317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghetxyc1/_remote_module_non_sriptable.py 2022-03-17T19:45:57.1710603Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:45:57.1726315Z 2022-03-17T19:45:57.1726701Z Running tests... 2022-03-17T19:45:57.1727184Z ---------------------------------------------------------------------- 2022-03-17T19:45:58.7050340Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:45:58.7467252Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68418 2022-03-17T19:45:58.7588310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68419 2022-03-17T19:45:58.7712357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68420 2022-03-17T19:45:58.7839313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68421 2022-03-17T19:45:59.5584514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bvvv970 2022-03-17T19:45:59.5586887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bvvv970/_remote_module_non_sriptable.py 2022-03-17T19:45:59.6058607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bro6vyj 2022-03-17T19:45:59.6061060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bro6vyj/_remote_module_non_sriptable.py 2022-03-17T19:45:59.6235781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpao5md0oa 2022-03-17T19:45:59.6239157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpao5md0oa/_remote_module_non_sriptable.py 2022-03-17T19:45:59.6392263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cxlczas 2022-03-17T19:45:59.6394815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cxlczas/_remote_module_non_sriptable.py 2022-03-17T19:45:59.9119587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:45:59.9589469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:45:59.9674562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:45:59.9829290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:02.8845705Z On WorkerInfo(id=1, name=worker1): 2022-03-17T19:46:02.8858227Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f39fa8c70bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f39fa8c2a8e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0x9b6 (0x7f3a044b87c6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f3a044baabf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf8 (0x7f3a044bc358 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f3a0468c66f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: + 0x29b1726 (0x7f39fd4b3726 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: + 0x29b1846 (0x7f39fd4b3846 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xd6 (0x7f3a04e76826 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: + 0x2790095 (0x7f3a05fa7095 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: + 0x2790829 (0x7f3a05fa7829 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x167 (0x7f3a04ea6e77 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: + 0x295a41 (0x7f3a10486a41 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #13: + 0x295d26 (0x7f3a10486d26 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #14: _PyMethodDef_RawFastCallDict + 0x24c (0x55974647d71c in /opt/conda/bin/python)\nframe #15: _PyObject_FastCallDict + 0x6e (0x55974644df5e in /opt/conda/bin/python)\nframe #16: + 0x1362e0 (0x55974646a2e0 in /opt/conda/bin/python)\nframe #17: + 0x1f529f (0x55974652929f in /opt/conda/bin/python)\nframe #18: PyNumber_Add + 0x41 (0x5597464893c1 in /opt/conda/bin/python)\nframe #19: _PyEval_EvalFrameDefault + 0x101b (0x5597464f7d7b in /opt/conda/bin/python)\nframe #20: _PyFunction_FastCallDict + 0x118 (0x55974646c138 in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x1d0d (0x5597464f8a6d in /opt/conda/bin/python)\nframe #22: _PyFunction_FastCallDict + 0x118 (0x55974646c138 in /opt/conda/bin/python)\nframe #23: + 0x8f2a34 (0x7f3a10ae3a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #24: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f3a10ae23dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7f3a10ae4db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x96 (0x7f3a10ae9216 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x10c (0x7f3a06ed954c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #28: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f3a10ae4ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #29: + 0x36bc0f3 (0x7f3a06ed30f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #30: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7f3a06ed3cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f3a06ece597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: + 0x36ea602 (0x7f3a06f01602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f39fa8b35eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #34: + 0xc9039 (0x7f3a1da3c039 in /opt/conda/lib/libstdc++.so.6)\nframe #35: + 0x76ba (0x7f3a3e4f16ba in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #36: clone + 0x6d (0x7f3a3e22751d in /lib/x86_64-linux-gnu/libc.so.6)\n') 2022-03-17T19:46:02.8866149Z Traceback (most recent call last): 2022-03-17T19:46:02.8866702Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-03-17T19:46:02.8867192Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-03-17T19:46:02.8867788Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5987, in _gpu_add_wrong_gpus 2022-03-17T19:46:02.8868202Z return x.cpu() + y.cuda() 2022-03-17T19:46:02.8868592Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 2022-03-17T19:46:02.8869112Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first): 2022-03-17T19:46:02.8869959Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f39fa8c70bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.8870929Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f39fa8c2a8e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.8871796Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0x9b6 (0x7f3a044b87c6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8872593Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f3a044baabf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8873479Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf8 (0x7f3a044bc358 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8874344Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f3a0468c66f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8875060Z frame #6: + 0x29b1726 (0x7f39fd4b3726 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:46:02.8875702Z frame #7: + 0x29b1846 (0x7f39fd4b3846 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:46:02.8876489Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xd6 (0x7f3a04e76826 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8877218Z frame #9: + 0x2790095 (0x7f3a05fa7095 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8877839Z frame #10: + 0x2790829 (0x7f3a05fa7829 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8878688Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x167 (0x7f3a04ea6e77 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8879392Z frame #12: + 0x295a41 (0x7f3a10486a41 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.8880017Z frame #13: + 0x295d26 (0x7f3a10486d26 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.8880470Z frame #14: _PyMethodDef_RawFastCallDict + 0x24c (0x55974647d71c in /opt/conda/bin/python) 2022-03-17T19:46:02.8880963Z frame #15: _PyObject_FastCallDict + 0x6e (0x55974644df5e in /opt/conda/bin/python) 2022-03-17T19:46:02.8881377Z frame #16: + 0x1362e0 (0x55974646a2e0 in /opt/conda/bin/python) 2022-03-17T19:46:02.8881744Z frame #17: + 0x1f529f (0x55974652929f in /opt/conda/bin/python) 2022-03-17T19:46:02.8882129Z frame #18: PyNumber_Add + 0x41 (0x5597464893c1 in /opt/conda/bin/python) 2022-03-17T19:46:02.8882508Z frame #19: _PyEval_EvalFrameDefault + 0x101b (0x5597464f7d7b in /opt/conda/bin/python) 2022-03-17T19:46:02.8882923Z frame #20: _PyFunction_FastCallDict + 0x118 (0x55974646c138 in /opt/conda/bin/python) 2022-03-17T19:46:02.8883334Z frame #21: _PyEval_EvalFrameDefault + 0x1d0d (0x5597464f8a6d in /opt/conda/bin/python) 2022-03-17T19:46:02.8883724Z frame #22: _PyFunction_FastCallDict + 0x118 (0x55974646c138 in /opt/conda/bin/python) 2022-03-17T19:46:02.8884325Z frame #23: + 0x8f2a34 (0x7f3a10ae3a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.8885090Z frame #24: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f3a10ae23dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.8886089Z frame #25: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7f3a10ae4db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.8887190Z frame #26: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x96 (0x7f3a10ae9216 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.8888394Z frame #27: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x10c (0x7f3a06ed954c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8889629Z frame #28: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f3a10ae4ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.8890512Z frame #29: + 0x36bc0f3 (0x7f3a06ed30f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8891434Z frame #30: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7f3a06ed3cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8892487Z frame #31: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f3a06ece597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8893244Z frame #32: + 0x36ea602 (0x7f3a06f01602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.8893981Z frame #33: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f39fa8b35eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.8894474Z frame #34: + 0xc9039 (0x7f3a1da3c039 in /opt/conda/lib/libstdc++.so.6) 2022-03-17T19:46:02.8894992Z frame #35: + 0x76ba (0x7f3a3e4f16ba in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-03-17T19:46:02.8895483Z frame #36: clone + 0x6d (0x7f3a3e22751d in /lib/x86_64-linux-gnu/libc.so.6) 2022-03-17T19:46:02.8895708Z 2022-03-17T19:46:02.8895727Z 2022-03-17T19:46:02.9050084Z On WorkerInfo(id=0, name=worker0): 2022-03-17T19:46:02.9063502Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f911de3a0bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f911de35a8e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0x9b6 (0x7f9127a2b7c6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f9127a2dabf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf8 (0x7f9127a2f358 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f9127bff66f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: + 0x29b1726 (0x7f9120a26726 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: + 0x29b1846 (0x7f9120a26846 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xd6 (0x7f91283e9826 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: + 0x2790095 (0x7f912951a095 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: + 0x2790829 (0x7f912951a829 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x167 (0x7f9128419e77 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: + 0x295a41 (0x7f91339f9a41 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #13: + 0x295d26 (0x7f91339f9d26 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #14: _PyMethodDef_RawFastCallDict + 0x24c (0x561724e4571c in /opt/conda/bin/python)\nframe #15: _PyObject_FastCallDict + 0x6e (0x561724e15f5e in /opt/conda/bin/python)\nframe #16: + 0x1362e0 (0x561724e322e0 in /opt/conda/bin/python)\nframe #17: + 0x1f529f (0x561724ef129f in /opt/conda/bin/python)\nframe #18: PyNumber_Add + 0x41 (0x561724e513c1 in /opt/conda/bin/python)\nframe #19: _PyEval_EvalFrameDefault + 0x101b (0x561724ebfd7b in /opt/conda/bin/python)\nframe #20: _PyFunction_FastCallDict + 0x118 (0x561724e34138 in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x1d0d (0x561724ec0a6d in /opt/conda/bin/python)\nframe #22: _PyFunction_FastCallDict + 0x118 (0x561724e34138 in /opt/conda/bin/python)\nframe #23: + 0x8f2a34 (0x7f9134056a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #24: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f91340553dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7f9134057db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x96 (0x7f913405c216 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x10c (0x7f912a44c54c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #28: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f9134057ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #29: + 0x36bc0f3 (0x7f912a4460f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #30: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7f912a446cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f912a441597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: + 0x36ea602 (0x7f912a474602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f911de265eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #34: + 0xc9039 (0x7f9140faf039 in /opt/conda/lib/libstdc++.so.6)\nframe #35: + 0x76ba (0x7f9161a646ba in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #36: clone + 0x6d (0x7f916179a51d in /lib/x86_64-linux-gnu/libc.so.6)\n') 2022-03-17T19:46:02.9071016Z Traceback (most recent call last): 2022-03-17T19:46:02.9071607Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-03-17T19:46:02.9072065Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-03-17T19:46:02.9072659Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5987, in _gpu_add_wrong_gpus 2022-03-17T19:46:02.9073071Z return x.cpu() + y.cuda() 2022-03-17T19:46:02.9073458Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 2022-03-17T19:46:02.9073968Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first): 2022-03-17T19:46:02.9074813Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f911de3a0bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9075780Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f911de35a8e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9076732Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0x9b6 (0x7f9127a2b7c6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9077523Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f9127a2dabf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9078403Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf8 (0x7f9127a2f358 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9079380Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f9127bff66f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9080113Z frame #6: + 0x29b1726 (0x7f9120a26726 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:46:02.9080752Z frame #7: + 0x29b1846 (0x7f9120a26846 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:46:02.9081533Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xd6 (0x7f91283e9826 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9082257Z frame #9: + 0x2790095 (0x7f912951a095 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9082871Z frame #10: + 0x2790829 (0x7f912951a829 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9083594Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x167 (0x7f9128419e77 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9084288Z frame #12: + 0x295a41 (0x7f91339f9a41 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9084927Z frame #13: + 0x295d26 (0x7f91339f9d26 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9085380Z frame #14: _PyMethodDef_RawFastCallDict + 0x24c (0x561724e4571c in /opt/conda/bin/python) 2022-03-17T19:46:02.9085798Z frame #15: _PyObject_FastCallDict + 0x6e (0x561724e15f5e in /opt/conda/bin/python) 2022-03-17T19:46:02.9086192Z frame #16: + 0x1362e0 (0x561724e322e0 in /opt/conda/bin/python) 2022-03-17T19:46:02.9086562Z frame #17: + 0x1f529f (0x561724ef129f in /opt/conda/bin/python) 2022-03-17T19:46:02.9086944Z frame #18: PyNumber_Add + 0x41 (0x561724e513c1 in /opt/conda/bin/python) 2022-03-17T19:46:02.9087324Z frame #19: _PyEval_EvalFrameDefault + 0x101b (0x561724ebfd7b in /opt/conda/bin/python) 2022-03-17T19:46:02.9087739Z frame #20: _PyFunction_FastCallDict + 0x118 (0x561724e34138 in /opt/conda/bin/python) 2022-03-17T19:46:02.9088155Z frame #21: _PyEval_EvalFrameDefault + 0x1d0d (0x561724ec0a6d in /opt/conda/bin/python) 2022-03-17T19:46:02.9088549Z frame #22: _PyFunction_FastCallDict + 0x118 (0x561724e34138 in /opt/conda/bin/python) 2022-03-17T19:46:02.9089140Z frame #23: + 0x8f2a34 (0x7f9134056a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9089889Z frame #24: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f91340553dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9090886Z frame #25: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7f9134057db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9091982Z frame #26: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x96 (0x7f913405c216 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9093257Z frame #27: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x10c (0x7f912a44c54c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9094538Z frame #28: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f9134057ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9095427Z frame #29: + 0x36bc0f3 (0x7f912a4460f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9096354Z frame #30: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7f912a446cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9097404Z frame #31: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f912a441597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9098158Z frame #32: + 0x36ea602 (0x7f912a474602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9098824Z frame #33: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f911de265eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9099309Z frame #34: + 0xc9039 (0x7f9140faf039 in /opt/conda/lib/libstdc++.so.6) 2022-03-17T19:46:02.9099829Z frame #35: + 0x76ba (0x7f9161a646ba in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-03-17T19:46:02.9100755Z frame #36: clone + 0x6d (0x7f916179a51d in /lib/x86_64-linux-gnu/libc.so.6) 2022-03-17T19:46:02.9100998Z 2022-03-17T19:46:02.9101018Z 2022-03-17T19:46:02.9197114Z On WorkerInfo(id=2, name=worker2): 2022-03-17T19:46:02.9209779Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f91a6b550bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f91a6b50a8e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0x9b6 (0x7f91b07467c6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f91b0748abf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf8 (0x7f91b074a358 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f91b091a66f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: + 0x29b1726 (0x7f91a9741726 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: + 0x29b1846 (0x7f91a9741846 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xd6 (0x7f91b1104826 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: + 0x2790095 (0x7f91b2235095 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: + 0x2790829 (0x7f91b2235829 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x167 (0x7f91b1134e77 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: + 0x295a41 (0x7f91bc714a41 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #13: + 0x295d26 (0x7f91bc714d26 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #14: _PyMethodDef_RawFastCallDict + 0x24c (0x55c8e104f71c in /opt/conda/bin/python)\nframe #15: _PyObject_FastCallDict + 0x6e (0x55c8e101ff5e in /opt/conda/bin/python)\nframe #16: + 0x1362e0 (0x55c8e103c2e0 in /opt/conda/bin/python)\nframe #17: + 0x1f529f (0x55c8e10fb29f in /opt/conda/bin/python)\nframe #18: PyNumber_Add + 0x41 (0x55c8e105b3c1 in /opt/conda/bin/python)\nframe #19: _PyEval_EvalFrameDefault + 0x101b (0x55c8e10c9d7b in /opt/conda/bin/python)\nframe #20: _PyFunction_FastCallDict + 0x118 (0x55c8e103e138 in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x1d0d (0x55c8e10caa6d in /opt/conda/bin/python)\nframe #22: _PyFunction_FastCallDict + 0x118 (0x55c8e103e138 in /opt/conda/bin/python)\nframe #23: + 0x8f2a34 (0x7f91bcd71a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #24: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f91bcd703dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7f91bcd72db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x96 (0x7f91bcd77216 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x10c (0x7f91b316754c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #28: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f91bcd72ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #29: + 0x36bc0f3 (0x7f91b31610f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #30: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7f91b3161cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f91b315c597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: + 0x36ea602 (0x7f91b318f602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f91a6b415eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #34: + 0xc9039 (0x7f91c9cca039 in /opt/conda/lib/libstdc++.so.6)\nframe #35: + 0x76ba (0x7f91ea77f6ba in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #36: clone + 0x6d (0x7f91ea4b551d in /lib/x86_64-linux-gnu/libc.so.6)\n') 2022-03-17T19:46:02.9218107Z Traceback (most recent call last): 2022-03-17T19:46:02.9218659Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-03-17T19:46:02.9219111Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-03-17T19:46:02.9219871Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5987, in _gpu_add_wrong_gpus 2022-03-17T19:46:02.9220630Z return x.cpu() + y.cuda() 2022-03-17T19:46:02.9221075Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 2022-03-17T19:46:02.9221600Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first): 2022-03-17T19:46:02.9222455Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f91a6b550bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9223462Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f91a6b50a8e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9224361Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0x9b6 (0x7f91b07467c6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9225127Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f91b0748abf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9226016Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf8 (0x7f91b074a358 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9226887Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f91b091a66f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9227584Z frame #6: + 0x29b1726 (0x7f91a9741726 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:46:02.9228203Z frame #7: + 0x29b1846 (0x7f91a9741846 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:46:02.9229011Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xd6 (0x7f91b1104826 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9229713Z frame #9: + 0x2790095 (0x7f91b2235095 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9230321Z frame #10: + 0x2790829 (0x7f91b2235829 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9231054Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x167 (0x7f91b1134e77 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9231730Z frame #12: + 0x295a41 (0x7f91bc714a41 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9232372Z frame #13: + 0x295d26 (0x7f91bc714d26 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9232847Z frame #14: _PyMethodDef_RawFastCallDict + 0x24c (0x55c8e104f71c in /opt/conda/bin/python) 2022-03-17T19:46:02.9233259Z frame #15: _PyObject_FastCallDict + 0x6e (0x55c8e101ff5e in /opt/conda/bin/python) 2022-03-17T19:46:02.9233778Z frame #16: + 0x1362e0 (0x55c8e103c2e0 in /opt/conda/bin/python) 2022-03-17T19:46:02.9234159Z frame #17: + 0x1f529f (0x55c8e10fb29f in /opt/conda/bin/python) 2022-03-17T19:46:02.9234541Z frame #18: PyNumber_Add + 0x41 (0x55c8e105b3c1 in /opt/conda/bin/python) 2022-03-17T19:46:02.9234938Z frame #19: _PyEval_EvalFrameDefault + 0x101b (0x55c8e10c9d7b in /opt/conda/bin/python) 2022-03-17T19:46:02.9235337Z frame #20: _PyFunction_FastCallDict + 0x118 (0x55c8e103e138 in /opt/conda/bin/python) 2022-03-17T19:46:02.9235828Z frame #21: _PyEval_EvalFrameDefault + 0x1d0d (0x55c8e10caa6d in /opt/conda/bin/python) 2022-03-17T19:46:02.9236259Z frame #22: _PyFunction_FastCallDict + 0x118 (0x55c8e103e138 in /opt/conda/bin/python) 2022-03-17T19:46:02.9236837Z frame #23: + 0x8f2a34 (0x7f91bcd71a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9237622Z frame #24: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f91bcd703dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9238596Z frame #25: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7f91bcd72db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9239708Z frame #26: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x96 (0x7f91bcd77216 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9240899Z frame #27: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x10c (0x7f91b316754c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9242163Z frame #28: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f91bcd72ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9243044Z frame #29: + 0x36bc0f3 (0x7f91b31610f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9243941Z frame #30: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7f91b3161cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9245006Z frame #31: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f91b315c597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9245778Z frame #32: + 0x36ea602 (0x7f91b318f602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9246425Z frame #33: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f91a6b415eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9246906Z frame #34: + 0xc9039 (0x7f91c9cca039 in /opt/conda/lib/libstdc++.so.6) 2022-03-17T19:46:02.9247420Z frame #35: + 0x76ba (0x7f91ea77f6ba in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-03-17T19:46:02.9247923Z frame #36: clone + 0x6d (0x7f91ea4b551d in /lib/x86_64-linux-gnu/libc.so.6) 2022-03-17T19:46:02.9248148Z 2022-03-17T19:46:02.9248159Z 2022-03-17T19:46:02.9257518Z On WorkerInfo(id=3, name=worker3): 2022-03-17T19:46:02.9287991Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f1aead0a0bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f1aead05a8e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0x9b6 (0x7f1af48fb7c6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f1af48fdabf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf8 (0x7f1af48ff358 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f1af4acf66f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: + 0x29b1726 (0x7f1aed8f6726 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: + 0x29b1846 (0x7f1aed8f6846 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xd6 (0x7f1af52b9826 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: + 0x2790095 (0x7f1af63ea095 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: + 0x2790829 (0x7f1af63ea829 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x167 (0x7f1af52e9e77 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: + 0x295a41 (0x7f1b008c9a41 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #13: + 0x295d26 (0x7f1b008c9d26 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #14: _PyMethodDef_RawFastCallDict + 0x24c (0x55c324b4371c in /opt/conda/bin/python)\nframe #15: _PyObject_FastCallDict + 0x6e (0x55c324b13f5e in /opt/conda/bin/python)\nframe #16: + 0x1362e0 (0x55c324b302e0 in /opt/conda/bin/python)\nframe #17: + 0x1f529f (0x55c324bef29f in /opt/conda/bin/python)\nframe #18: PyNumber_Add + 0x41 (0x55c324b4f3c1 in /opt/conda/bin/python)\nframe #19: _PyEval_EvalFrameDefault + 0x101b (0x55c324bbdd7b in /opt/conda/bin/python)\nframe #20: _PyFunction_FastCallDict + 0x118 (0x55c324b32138 in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x1d0d (0x55c324bbea6d in /opt/conda/bin/python)\nframe #22: _PyFunction_FastCallDict + 0x118 (0x55c324b32138 in /opt/conda/bin/python)\nframe #23: + 0x8f2a34 (0x7f1b00f26a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #24: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f1b00f253dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7f1b00f27db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x96 (0x7f1b00f2c216 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x10c (0x7f1af731c54c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #28: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f1b00f27ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)\nframe #29: + 0x36bc0f3 (0x7f1af73160f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #30: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7f1af7316cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f1af7311597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: + 0x36ea602 (0x7f1af7344602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f1aeacf65eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)\nframe #34: + 0xc9039 (0x7f1b0de7f039 in /opt/conda/lib/libstdc++.so.6)\nframe #35: + 0x76ba (0x7f1b2e9346ba in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #36: clone + 0x6d (0x7f1b2e66a51d in /lib/x86_64-linux-gnu/libc.so.6)\n') 2022-03-17T19:46:02.9306894Z Traceback (most recent call last): 2022-03-17T19:46:02.9308201Z File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function 2022-03-17T19:46:02.9309252Z result = python_udf.func(*python_udf.args, **python_udf.kwargs) 2022-03-17T19:46:02.9310790Z File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5987, in _gpu_add_wrong_gpus 2022-03-17T19:46:02.9311769Z return x.cpu() + y.cuda() 2022-03-17T19:46:02.9312678Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! 2022-03-17T19:46:02.9313983Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first): 2022-03-17T19:46:02.9316039Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x6b (0x7f1aead0a0bb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9318410Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xce (0x7f1aead05a8e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9320585Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0x9b6 (0x7f1af48fb7c6 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9322522Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f1af48fdabf in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9324785Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf8 (0x7f1af48ff358 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9326934Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2f (0x7f1af4acf66f in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9328871Z frame #6: + 0x29b1726 (0x7f1aed8f6726 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:46:02.9330441Z frame #7: + 0x29b1846 (0x7f1aed8f6846 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so) 2022-03-17T19:46:02.9332385Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0xd6 (0x7f1af52b9826 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9334260Z frame #9: + 0x2790095 (0x7f1af63ea095 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9335805Z frame #10: + 0x2790829 (0x7f1af63ea829 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9337580Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x167 (0x7f1af52e9e77 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9339279Z frame #12: + 0x295a41 (0x7f1b008c9a41 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9341030Z frame #13: + 0x295d26 (0x7f1b008c9d26 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9342189Z frame #14: _PyMethodDef_RawFastCallDict + 0x24c (0x55c324b4371c in /opt/conda/bin/python) 2022-03-17T19:46:02.9343199Z frame #15: _PyObject_FastCallDict + 0x6e (0x55c324b13f5e in /opt/conda/bin/python) 2022-03-17T19:46:02.9344128Z frame #16: + 0x1362e0 (0x55c324b302e0 in /opt/conda/bin/python) 2022-03-17T19:46:02.9345065Z frame #17: + 0x1f529f (0x55c324bef29f in /opt/conda/bin/python) 2022-03-17T19:46:02.9345993Z frame #18: PyNumber_Add + 0x41 (0x55c324b4f3c1 in /opt/conda/bin/python) 2022-03-17T19:46:02.9346925Z frame #19: _PyEval_EvalFrameDefault + 0x101b (0x55c324bbdd7b in /opt/conda/bin/python) 2022-03-17T19:46:02.9347930Z frame #20: _PyFunction_FastCallDict + 0x118 (0x55c324b32138 in /opt/conda/bin/python) 2022-03-17T19:46:02.9348902Z frame #21: _PyEval_EvalFrameDefault + 0x1d0d (0x55c324bbea6d in /opt/conda/bin/python) 2022-03-17T19:46:02.9349905Z frame #22: _PyFunction_FastCallDict + 0x118 (0x55c324b32138 in /opt/conda/bin/python) 2022-03-17T19:46:02.9351352Z frame #23: + 0x8f2a34 (0x7f1b00f26a34 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9353221Z frame #24: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f1b00f253dd in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9355665Z frame #25: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector >, bool) const + 0x83 (0x7f1b00f27db3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9358390Z frame #26: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector >) const + 0x96 (0x7f1b00f2c216 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9361330Z frame #27: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x10c (0x7f1af731c54c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9364452Z frame #28: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector >) const + 0x65 (0x7f1b00f27ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so) 2022-03-17T19:46:02.9366782Z frame #29: + 0x36bc0f3 (0x7f1af73160f3 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9369052Z frame #30: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector >) const + 0x518 (0x7f1af7316cc8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9371697Z frame #31: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector >) const + 0x57 (0x7f1af7311597 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9373611Z frame #32: + 0x36ea602 (0x7f1af7344602 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so) 2022-03-17T19:46:02.9375232Z frame #33: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f1aeacf65eb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so) 2022-03-17T19:46:02.9376383Z frame #34: + 0xc9039 (0x7f1b0de7f039 in /opt/conda/lib/libstdc++.so.6) 2022-03-17T19:46:02.9377662Z frame #35: + 0x76ba (0x7f1b2e9346ba in /lib/x86_64-linux-gnu/libpthread.so.0) 2022-03-17T19:46:02.9378830Z frame #36: clone + 0x6d (0x7f1b2e66a51d in /lib/x86_64-linux-gnu/libc.so.6) 2022-03-17T19:46:02.9379315Z 2022-03-17T19:46:02.9379347Z 2022-03-17T19:46:03.3948473Z ok (6.222s) 2022-03-17T19:46:03.3948655Z 2022-03-17T19:46:03.3949044Z ---------------------------------------------------------------------- 2022-03-17T19:46:03.3949454Z Ran 1 test in 6.222s 2022-03-17T19:46:03.3949600Z 2022-03-17T19:46:03.3949692Z OK 2022-03-17T19:46:03.3949831Z 2022-03-17T19:46:03.3949872Z Generating XML reports... 2022-03-17T19:46:03.3994516Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194557.xml 2022-03-17T19:46:04.4416594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt94q49f 2022-03-17T19:46:04.4417513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt94q49f/_remote_module_non_sriptable.py 2022-03-17T19:46:04.7981503Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:46:04.7999098Z 2022-03-17T19:46:04.7999439Z Running tests... 2022-03-17T19:46:04.7999874Z ---------------------------------------------------------------------- 2022-03-17T19:46:06.3363533Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:46:06.3798330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68771 2022-03-17T19:46:06.3918064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68772 2022-03-17T19:46:06.4042434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68773 2022-03-17T19:46:06.4169331Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68774 2022-03-17T19:46:07.2048682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn6_n301 2022-03-17T19:46:07.2049649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn6_n301/_remote_module_non_sriptable.py 2022-03-17T19:46:07.2484445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtv3v3z7 2022-03-17T19:46:07.2486965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtv3v3z7/_remote_module_non_sriptable.py 2022-03-17T19:46:07.2528321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcp156w_ 2022-03-17T19:46:07.2530747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcp156w_/_remote_module_non_sriptable.py 2022-03-17T19:46:07.2537880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6i9la0g 2022-03-17T19:46:07.2541130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6i9la0g/_remote_module_non_sriptable.py 2022-03-17T19:46:07.5534240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:07.5903085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:07.5926880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:07.5959120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:07.9217111Z ok (3.121s) 2022-03-17T19:46:07.9217312Z 2022-03-17T19:46:07.9217933Z ---------------------------------------------------------------------- 2022-03-17T19:46:07.9218316Z Ran 1 test in 3.122s 2022-03-17T19:46:07.9218448Z 2022-03-17T19:46:07.9218554Z OK 2022-03-17T19:46:07.9218690Z 2022-03-17T19:46:07.9218817Z Generating XML reports... 2022-03-17T19:46:07.9262600Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194604.xml 2022-03-17T19:46:08.9772235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb68yy6ls 2022-03-17T19:46:08.9773064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb68yy6ls/_remote_module_non_sriptable.py 2022-03-17T19:46:09.3317672Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:46:09.3333694Z 2022-03-17T19:46:09.3334148Z Running tests... 2022-03-17T19:46:09.3334660Z ---------------------------------------------------------------------- 2022-03-17T19:46:10.8603302Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:46:10.9036630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68964 2022-03-17T19:46:10.9157914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68965 2022-03-17T19:46:10.9283682Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68966 2022-03-17T19:46:10.9411913Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68967 2022-03-17T19:46:11.7156469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hl_xt0t 2022-03-17T19:46:11.7157554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hl_xt0t/_remote_module_non_sriptable.py 2022-03-17T19:46:11.7205390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu59c99um 2022-03-17T19:46:11.7208256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu59c99um/_remote_module_non_sriptable.py 2022-03-17T19:46:11.7326074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz85exbxi 2022-03-17T19:46:11.7328173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz85exbxi/_remote_module_non_sriptable.py 2022-03-17T19:46:11.7607362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ijdiowd 2022-03-17T19:46:11.7609509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ijdiowd/_remote_module_non_sriptable.py 2022-03-17T19:46:12.0608780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:12.0652144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:12.0794326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:12.1089179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:12.3458266Z ok (3.012s) 2022-03-17T19:46:12.3458490Z 2022-03-17T19:46:12.3458870Z ---------------------------------------------------------------------- 2022-03-17T19:46:12.3459224Z Ran 1 test in 3.012s 2022-03-17T19:46:12.3459390Z 2022-03-17T19:46:12.3459815Z OK 2022-03-17T19:46:12.3459887Z 2022-03-17T19:46:12.3459988Z Generating XML reports... 2022-03-17T19:46:12.3502800Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194609.xml 2022-03-17T19:46:13.3953345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu26h6x42 2022-03-17T19:46:13.3954096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu26h6x42/_remote_module_non_sriptable.py 2022-03-17T19:46:13.7510163Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:46:13.7525744Z 2022-03-17T19:46:13.7526087Z Running tests... 2022-03-17T19:46:13.7526555Z ---------------------------------------------------------------------- 2022-03-17T19:46:15.2774723Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:46:15.3191153Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69157 2022-03-17T19:46:15.3311449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69158 2022-03-17T19:46:15.3434468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69159 2022-03-17T19:46:15.3559484Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69160 2022-03-17T19:46:16.1746244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcrhwbct 2022-03-17T19:46:16.1747291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcrhwbct/_remote_module_non_sriptable.py 2022-03-17T19:46:16.1986351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpii2mxhnx 2022-03-17T19:46:16.1988857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpii2mxhnx/_remote_module_non_sriptable.py 2022-03-17T19:46:16.2406728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptaeagq16 2022-03-17T19:46:16.2409390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptaeagq16/_remote_module_non_sriptable.py 2022-03-17T19:46:16.2411724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mwcwxga 2022-03-17T19:46:16.2414490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mwcwxga/_remote_module_non_sriptable.py 2022-03-17T19:46:16.5176285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:16.5495765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:16.5879430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:16.5915538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:21.5685754Z ok (7.816s) 2022-03-17T19:46:21.5685943Z 2022-03-17T19:46:21.5686358Z ---------------------------------------------------------------------- 2022-03-17T19:46:21.5686745Z Ran 1 test in 7.816s 2022-03-17T19:46:21.5686923Z 2022-03-17T19:46:21.5686988Z OK 2022-03-17T19:46:21.5687130Z 2022-03-17T19:46:21.5687273Z Generating XML reports... 2022-03-17T19:46:21.5731671Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194613.xml 2022-03-17T19:46:22.6179185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcok6n5co 2022-03-17T19:46:22.6180529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcok6n5co/_remote_module_non_sriptable.py 2022-03-17T19:46:22.9687485Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:46:22.9702497Z 2022-03-17T19:46:22.9702724Z Running tests... 2022-03-17T19:46:22.9703165Z ---------------------------------------------------------------------- 2022-03-17T19:46:24.5009946Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:46:24.5427117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69381 2022-03-17T19:46:24.5546766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69382 2022-03-17T19:46:24.5671678Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69383 2022-03-17T19:46:24.5800428Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69384 2022-03-17T19:46:25.4199622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbc0h1nb 2022-03-17T19:46:25.4200888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbc0h1nb/_remote_module_non_sriptable.py 2022-03-17T19:46:25.4749968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxiswn30 2022-03-17T19:46:25.4751236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxiswn30/_remote_module_non_sriptable.py 2022-03-17T19:46:25.4827751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf29l73ky 2022-03-17T19:46:25.4830681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf29l73ky/_remote_module_non_sriptable.py 2022-03-17T19:46:25.5042008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hrrrcx2 2022-03-17T19:46:25.5044778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hrrrcx2/_remote_module_non_sriptable.py 2022-03-17T19:46:25.7669256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:25.8214987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:25.8363108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:25.8456426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:32.2953093Z ok (9.325s) 2022-03-17T19:46:32.2953305Z 2022-03-17T19:46:32.2953680Z ---------------------------------------------------------------------- 2022-03-17T19:46:32.2954060Z Ran 1 test in 9.325s 2022-03-17T19:46:32.2954222Z 2022-03-17T19:46:32.2954294Z OK 2022-03-17T19:46:32.2954436Z 2022-03-17T19:46:32.2954575Z Generating XML reports... 2022-03-17T19:46:32.2998387Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194622.xml 2022-03-17T19:46:33.3436048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8hb14gj 2022-03-17T19:46:33.3437202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8hb14gj/_remote_module_non_sriptable.py 2022-03-17T19:46:33.6970141Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:46:33.6985897Z 2022-03-17T19:46:33.6986170Z Running tests... 2022-03-17T19:46:33.6986565Z ---------------------------------------------------------------------- 2022-03-17T19:46:35.2353110Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:46:35.2775202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69606 2022-03-17T19:46:35.2897235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69607 2022-03-17T19:46:35.3022485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69608 2022-03-17T19:46:35.3153212Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69609 2022-03-17T19:46:36.1215584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8dzfs2b 2022-03-17T19:46:36.1216147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57jchaer 2022-03-17T19:46:36.1216881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8dzfs2b/_remote_module_non_sriptable.py 2022-03-17T19:46:36.1217641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57jchaer/_remote_module_non_sriptable.py 2022-03-17T19:46:36.1796603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6om8nl9 2022-03-17T19:46:36.1799594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6om8nl9/_remote_module_non_sriptable.py 2022-03-17T19:46:36.1877128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp00j6tmx 2022-03-17T19:46:36.1879919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp00j6tmx/_remote_module_non_sriptable.py 2022-03-17T19:46:36.4726330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:36.4741109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:36.5315997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:36.5364455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:42.8300130Z ok (9.131s) 2022-03-17T19:46:42.8300631Z 2022-03-17T19:46:42.8300989Z ---------------------------------------------------------------------- 2022-03-17T19:46:42.8301328Z Ran 1 test in 9.131s 2022-03-17T19:46:42.8301497Z 2022-03-17T19:46:42.8303287Z OK 2022-03-17T19:46:42.8303493Z 2022-03-17T19:46:42.8303636Z Generating XML reports... 2022-03-17T19:46:42.8344273Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194633.xml 2022-03-17T19:46:43.8826916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgf_u56l 2022-03-17T19:46:43.8827976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgf_u56l/_remote_module_non_sriptable.py 2022-03-17T19:46:44.2462665Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:46:44.2478764Z 2022-03-17T19:46:44.2479093Z Running tests... 2022-03-17T19:46:44.2479547Z ---------------------------------------------------------------------- 2022-03-17T19:46:45.7785396Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:46:45.8206264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69831 2022-03-17T19:46:45.8327564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69832 2022-03-17T19:46:45.8453105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69833 2022-03-17T19:46:45.8583096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69834 2022-03-17T19:46:46.6852958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hq0uktm 2022-03-17T19:46:46.6854218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hq0uktm/_remote_module_non_sriptable.py 2022-03-17T19:46:46.7175697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46qmc2j4 2022-03-17T19:46:46.7178240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46qmc2j4/_remote_module_non_sriptable.py 2022-03-17T19:46:46.7239659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazm4x37c 2022-03-17T19:46:46.7242708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazm4x37c/_remote_module_non_sriptable.py 2022-03-17T19:46:46.7281135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkwe43n0 2022-03-17T19:46:46.7283855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkwe43n0/_remote_module_non_sriptable.py 2022-03-17T19:46:47.0283157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:47.0613595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:46:47.0704104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:47.0773224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:51.8712267Z ok (7.623s) 2022-03-17T19:46:51.8712792Z 2022-03-17T19:46:51.8713456Z ---------------------------------------------------------------------- 2022-03-17T19:46:51.8714001Z Ran 1 test in 7.623s 2022-03-17T19:46:51.8714279Z 2022-03-17T19:46:51.8714454Z OK 2022-03-17T19:46:51.8714689Z 2022-03-17T19:46:51.8714933Z Generating XML reports... 2022-03-17T19:46:51.8760802Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194644.xml 2022-03-17T19:46:52.9208535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wmoxmsz 2022-03-17T19:46:52.9209427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wmoxmsz/_remote_module_non_sriptable.py 2022-03-17T19:46:53.2802436Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:46:53.2817657Z 2022-03-17T19:46:53.2817941Z Running tests... 2022-03-17T19:46:53.2818361Z ---------------------------------------------------------------------- 2022-03-17T19:46:54.8106727Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:46:54.8528703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70055 2022-03-17T19:46:54.8649984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70056 2022-03-17T19:46:54.8773236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70057 2022-03-17T19:46:54.8900725Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70058 2022-03-17T19:46:55.7005345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl53rtxwn 2022-03-17T19:46:55.7006232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl53rtxwn/_remote_module_non_sriptable.py 2022-03-17T19:46:55.7053130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyhw6nm3 2022-03-17T19:46:55.7055878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyhw6nm3/_remote_module_non_sriptable.py 2022-03-17T19:46:55.7556331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplp9gsl0a 2022-03-17T19:46:55.7558640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplp9gsl0a/_remote_module_non_sriptable.py 2022-03-17T19:46:55.7571282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7a23eb9 2022-03-17T19:46:55.7573999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7a23eb9/_remote_module_non_sriptable.py 2022-03-17T19:46:56.0507727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:46:56.0540194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:46:56.1011267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:46:56.1081134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:09.6170773Z ok (16.335s) 2022-03-17T19:47:09.6171016Z 2022-03-17T19:47:09.6171428Z ---------------------------------------------------------------------- 2022-03-17T19:47:09.6171760Z Ran 1 test in 16.335s 2022-03-17T19:47:09.6171905Z 2022-03-17T19:47:09.6171988Z OK 2022-03-17T19:47:09.6172125Z 2022-03-17T19:47:09.6172259Z Generating XML reports... 2022-03-17T19:47:09.6214771Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194653.xml 2022-03-17T19:47:10.6702988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt9pde5o 2022-03-17T19:47:10.6704522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt9pde5o/_remote_module_non_sriptable.py 2022-03-17T19:47:11.0292089Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:47:11.0307768Z 2022-03-17T19:47:11.0307905Z Running tests... 2022-03-17T19:47:11.0308347Z ---------------------------------------------------------------------- 2022-03-17T19:47:12.5673952Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:47:12.6094921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70408 2022-03-17T19:47:12.6216971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70409 2022-03-17T19:47:12.6340332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70410 2022-03-17T19:47:12.6468598Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70411 2022-03-17T19:47:13.4769043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzocqli0z 2022-03-17T19:47:13.4771209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzocqli0z/_remote_module_non_sriptable.py 2022-03-17T19:47:13.4919305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicwcq53v 2022-03-17T19:47:13.4922025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicwcq53v/_remote_module_non_sriptable.py 2022-03-17T19:47:13.4926795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps90ksp72 2022-03-17T19:47:13.4929913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps90ksp72/_remote_module_non_sriptable.py 2022-03-17T19:47:13.5283253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39883lb7 2022-03-17T19:47:13.5285905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39883lb7/_remote_module_non_sriptable.py 2022-03-17T19:47:13.8291492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:13.8405219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:13.8489085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:47:13.8746674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:47:29.2770729Z ok (18.246s) 2022-03-17T19:47:29.2770924Z 2022-03-17T19:47:29.2771702Z ---------------------------------------------------------------------- 2022-03-17T19:47:29.2772056Z Ran 1 test in 18.246s 2022-03-17T19:47:29.2772233Z 2022-03-17T19:47:29.2772321Z OK 2022-03-17T19:47:29.2772489Z 2022-03-17T19:47:29.2772591Z Generating XML reports... 2022-03-17T19:47:29.2817920Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194711.xml 2022-03-17T19:47:30.3188834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvy25wqf0 2022-03-17T19:47:30.3189689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvy25wqf0/_remote_module_non_sriptable.py 2022-03-17T19:47:30.6747752Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:47:30.6762550Z 2022-03-17T19:47:30.6762656Z Running tests... 2022-03-17T19:47:30.6763601Z ---------------------------------------------------------------------- 2022-03-17T19:47:32.2126817Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:47:32.2547996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70767 2022-03-17T19:47:32.2671250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70768 2022-03-17T19:47:32.2795252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70769 2022-03-17T19:47:32.2921469Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70770 2022-03-17T19:47:33.0775795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gz9fqcm 2022-03-17T19:47:33.0777099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gz9fqcm/_remote_module_non_sriptable.py 2022-03-17T19:47:33.1142823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47z1yzj2 2022-03-17T19:47:33.1146319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47z1yzj2/_remote_module_non_sriptable.py 2022-03-17T19:47:33.1379701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpm8fqmf 2022-03-17T19:47:33.1382818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpm8fqmf/_remote_module_non_sriptable.py 2022-03-17T19:47:33.1390049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlg8t32_ 2022-03-17T19:47:33.1392843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlg8t32_/_remote_module_non_sriptable.py 2022-03-17T19:47:33.4304637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:33.4596558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:33.4839327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:47:33.4848497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:47:46.7189086Z ok (16.042s) 2022-03-17T19:47:46.7189454Z 2022-03-17T19:47:46.7190110Z ---------------------------------------------------------------------- 2022-03-17T19:47:46.7190773Z Ran 1 test in 16.042s 2022-03-17T19:47:46.7191083Z 2022-03-17T19:47:46.7192915Z OK 2022-03-17T19:47:46.7193120Z 2022-03-17T19:47:46.7193290Z Generating XML reports... 2022-03-17T19:47:46.7234221Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194730.xml 2022-03-17T19:47:47.7649505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcy_h1yo 2022-03-17T19:47:47.7650601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcy_h1yo/_remote_module_non_sriptable.py 2022-03-17T19:47:48.1208724Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:47:48.1225837Z 2022-03-17T19:47:48.1226274Z Running tests... 2022-03-17T19:47:48.1226790Z ---------------------------------------------------------------------- 2022-03-17T19:47:49.6565537Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:47:49.6988262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71120 2022-03-17T19:47:49.7108995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71121 2022-03-17T19:47:49.7233946Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71122 2022-03-17T19:47:49.7364470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71123 2022-03-17T19:47:50.5714148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhkibp61 2022-03-17T19:47:50.5715041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhkibp61/_remote_module_non_sriptable.py 2022-03-17T19:47:50.6101421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq6iz6wa 2022-03-17T19:47:50.6104126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq6iz6wa/_remote_module_non_sriptable.py 2022-03-17T19:47:50.6147321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_5bxnxf 2022-03-17T19:47:50.6150218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_5bxnxf/_remote_module_non_sriptable.py 2022-03-17T19:47:50.6158294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvrmb68_ 2022-03-17T19:47:50.6161220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvrmb68_/_remote_module_non_sriptable.py 2022-03-17T19:47:50.9235092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:47:50.9570542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:47:50.9609227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:47:50.9642301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:48:06.5680318Z ok (18.445s) 2022-03-17T19:48:06.5681561Z 2022-03-17T19:48:06.5681989Z ---------------------------------------------------------------------- 2022-03-17T19:48:06.5682381Z Ran 1 test in 18.445s 2022-03-17T19:48:06.5682549Z 2022-03-17T19:48:06.5682641Z OK 2022-03-17T19:48:06.5682763Z 2022-03-17T19:48:06.5682896Z Generating XML reports... 2022-03-17T19:48:06.5723594Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194748.xml 2022-03-17T19:48:07.6103792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_sds0ur 2022-03-17T19:48:07.6104674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_sds0ur/_remote_module_non_sriptable.py 2022-03-17T19:48:07.9638762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:48:07.9653724Z 2022-03-17T19:48:07.9654182Z Running tests... 2022-03-17T19:48:07.9654671Z ---------------------------------------------------------------------- 2022-03-17T19:48:09.4931390Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:48:09.5352623Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71479 2022-03-17T19:48:09.5476858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71480 2022-03-17T19:48:09.5607130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71481 2022-03-17T19:48:09.5735634Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71482 2022-03-17T19:48:10.3551762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m25fxgn 2022-03-17T19:48:10.3552599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m25fxgn/_remote_module_non_sriptable.py 2022-03-17T19:48:10.3761255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5o45biq 2022-03-17T19:48:10.3764112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5o45biq/_remote_module_non_sriptable.py 2022-03-17T19:48:10.3770415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpju2ixbi3 2022-03-17T19:48:10.3773518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpju2ixbi3/_remote_module_non_sriptable.py 2022-03-17T19:48:10.4150275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuesh7yf 2022-03-17T19:48:10.4153155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuesh7yf/_remote_module_non_sriptable.py 2022-03-17T19:48:10.7052932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:10.7196911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:48:10.7255675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:10.7590604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:48:24.1014831Z ok (16.136s) 2022-03-17T19:48:24.1015098Z 2022-03-17T19:48:24.1015515Z ---------------------------------------------------------------------- 2022-03-17T19:48:24.1015830Z Ran 1 test in 16.136s 2022-03-17T19:48:24.1016337Z 2022-03-17T19:48:24.1016412Z OK 2022-03-17T19:48:24.1016549Z 2022-03-17T19:48:24.1016684Z Generating XML reports... 2022-03-17T19:48:24.1062654Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194807.xml 2022-03-17T19:48:25.1461486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6f1oqr0 2022-03-17T19:48:25.1462596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6f1oqr0/_remote_module_non_sriptable.py 2022-03-17T19:48:25.5079420Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:48:25.5094872Z 2022-03-17T19:48:25.5095239Z Running tests... 2022-03-17T19:48:25.5095703Z ---------------------------------------------------------------------- 2022-03-17T19:48:27.0425668Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:48:27.0852013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71832 2022-03-17T19:48:27.0977740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71833 2022-03-17T19:48:27.1101863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71834 2022-03-17T19:48:27.1229622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71835 2022-03-17T19:48:27.9567153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k140inh 2022-03-17T19:48:27.9569209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k140inh/_remote_module_non_sriptable.py 2022-03-17T19:48:27.9588194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3vitpsl 2022-03-17T19:48:27.9591236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3vitpsl/_remote_module_non_sriptable.py 2022-03-17T19:48:27.9777664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vdm41g0 2022-03-17T19:48:27.9780323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vdm41g0/_remote_module_non_sriptable.py 2022-03-17T19:48:27.9811289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85bydi79 2022-03-17T19:48:27.9814309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85bydi79/_remote_module_non_sriptable.py 2022-03-17T19:48:28.3018656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:28.3042240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:28.3303483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:48:28.3324318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:48:40.4492026Z ok (14.939s) 2022-03-17T19:48:40.4492263Z 2022-03-17T19:48:40.4492909Z ---------------------------------------------------------------------- 2022-03-17T19:48:40.4493274Z Ran 1 test in 14.940s 2022-03-17T19:48:40.4493446Z 2022-03-17T19:48:40.4493508Z OK 2022-03-17T19:48:40.4493646Z 2022-03-17T19:48:40.4493788Z Generating XML reports... 2022-03-17T19:48:40.4536218Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194825.xml 2022-03-17T19:48:41.4963206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn8tj9s9 2022-03-17T19:48:41.4964257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn8tj9s9/_remote_module_non_sriptable.py 2022-03-17T19:48:41.8445112Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:48:41.8459931Z 2022-03-17T19:48:41.8460083Z Running tests... 2022-03-17T19:48:41.8460540Z ---------------------------------------------------------------------- 2022-03-17T19:48:43.3531192Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:48:43.3945493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72184 2022-03-17T19:48:43.4067911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72185 2022-03-17T19:48:43.4190622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72186 2022-03-17T19:48:43.4318002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72187 2022-03-17T19:48:44.2955885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwfj5rnd 2022-03-17T19:48:44.2956703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwfj5rnd/_remote_module_non_sriptable.py 2022-03-17T19:48:44.2977670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuu3wzsbv 2022-03-17T19:48:44.2978240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4kfckx6 2022-03-17T19:48:44.2981457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuu3wzsbv/_remote_module_non_sriptable.py 2022-03-17T19:48:44.2982055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4kfckx6/_remote_module_non_sriptable.py 2022-03-17T19:48:44.3000496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1rjko1a 2022-03-17T19:48:44.3003350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1rjko1a/_remote_module_non_sriptable.py 2022-03-17T19:48:44.6367635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:48:44.6430256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:48:44.6504650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:48:44.6551417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:48:56.7565294Z ok (14.910s) 2022-03-17T19:48:56.7565491Z 2022-03-17T19:48:56.7565943Z ---------------------------------------------------------------------- 2022-03-17T19:48:56.7566264Z Ran 1 test in 14.910s 2022-03-17T19:48:56.7566431Z 2022-03-17T19:48:56.7566504Z OK 2022-03-17T19:48:56.7566583Z 2022-03-17T19:48:56.7566735Z Generating XML reports... 2022-03-17T19:48:56.7610212Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194841.xml 2022-03-17T19:48:57.7969941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdhcscyz 2022-03-17T19:48:57.7970944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdhcscyz/_remote_module_non_sriptable.py 2022-03-17T19:48:58.1504999Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:48:58.1520300Z 2022-03-17T19:48:58.1520540Z Running tests... 2022-03-17T19:48:58.1520990Z ---------------------------------------------------------------------- 2022-03-17T19:48:59.6652726Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:48:59.7069289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72539 2022-03-17T19:48:59.7189414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72540 2022-03-17T19:48:59.7312693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72541 2022-03-17T19:48:59.7438238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72542 2022-03-17T19:49:00.6043710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpluoxks7d 2022-03-17T19:49:00.6044844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpluoxks7d/_remote_module_non_sriptable.py 2022-03-17T19:49:00.6542115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpguwblqna 2022-03-17T19:49:00.6543810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpguwblqna/_remote_module_non_sriptable.py 2022-03-17T19:49:00.6611371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sda_pox 2022-03-17T19:49:00.6614054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sda_pox/_remote_module_non_sriptable.py 2022-03-17T19:49:00.6754830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzuyx7ze 2022-03-17T19:49:00.6758649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzuyx7ze/_remote_module_non_sriptable.py 2022-03-17T19:49:00.9545568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:49:01.0046833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:49:01.0058241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:49:01.0305360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:49:12.8680253Z ok (14.716s) 2022-03-17T19:49:12.8680407Z 2022-03-17T19:49:12.8680823Z ---------------------------------------------------------------------- 2022-03-17T19:49:12.8681202Z Ran 1 test in 14.716s 2022-03-17T19:49:12.8681373Z 2022-03-17T19:49:12.8681396Z OK 2022-03-17T19:49:12.8681532Z 2022-03-17T19:49:12.8681653Z Generating XML reports... 2022-03-17T19:49:12.8725565Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194858.xml 2022-03-17T19:49:13.9177754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdoamg6w0 2022-03-17T19:49:13.9178664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdoamg6w0/_remote_module_non_sriptable.py 2022-03-17T19:49:14.2712551Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:49:14.2727730Z 2022-03-17T19:49:14.2727867Z Running tests... 2022-03-17T19:49:14.2728538Z ---------------------------------------------------------------------- 2022-03-17T19:49:15.8131425Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:49:15.8552676Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72894 2022-03-17T19:49:15.8674277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72895 2022-03-17T19:49:15.8801845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72896 2022-03-17T19:49:15.8926019Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72897 2022-03-17T19:49:16.7262635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbdtsa3l 2022-03-17T19:49:16.7264024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbdtsa3l/_remote_module_non_sriptable.py 2022-03-17T19:49:16.7264919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3qv409x 2022-03-17T19:49:16.7267977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3qv409x/_remote_module_non_sriptable.py 2022-03-17T19:49:16.7353476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5j1kwc4 2022-03-17T19:49:16.7356414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5j1kwc4/_remote_module_non_sriptable.py 2022-03-17T19:49:16.7407202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhcpq322 2022-03-17T19:49:16.7410129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhcpq322/_remote_module_non_sriptable.py 2022-03-17T19:49:17.0702567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:49:17.0716436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:49:17.0859395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:49:17.0887742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:49:28.8166521Z ok (14.543s) 2022-03-17T19:49:28.8166761Z 2022-03-17T19:49:28.8167133Z ---------------------------------------------------------------------- 2022-03-17T19:49:28.8167482Z Ran 1 test in 14.544s 2022-03-17T19:49:28.8167658Z 2022-03-17T19:49:28.8167753Z OK 2022-03-17T19:49:28.8167871Z 2022-03-17T19:49:28.8168008Z Generating XML reports... 2022-03-17T19:49:28.8212766Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194914.xml 2022-03-17T19:49:29.8674510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu96gjlvi 2022-03-17T19:49:29.8675379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu96gjlvi/_remote_module_non_sriptable.py 2022-03-17T19:49:30.2237019Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:49:30.2253118Z 2022-03-17T19:49:30.2253470Z Running tests... 2022-03-17T19:49:30.2253938Z ---------------------------------------------------------------------- 2022-03-17T19:49:31.7673502Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:49:31.8097164Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73246 2022-03-17T19:49:31.8221558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73247 2022-03-17T19:49:31.8348242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73248 2022-03-17T19:49:31.8473615Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73249 2022-03-17T19:49:32.6436354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8gf0pdd 2022-03-17T19:49:32.6437582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8gf0pdd/_remote_module_non_sriptable.py 2022-03-17T19:49:32.6849373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2olp_8y 2022-03-17T19:49:32.6851594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2olp_8y/_remote_module_non_sriptable.py 2022-03-17T19:49:32.6893375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmuuk4vf2 2022-03-17T19:49:32.6896090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmuuk4vf2/_remote_module_non_sriptable.py 2022-03-17T19:49:32.7246271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdwlcx97 2022-03-17T19:49:32.7248703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdwlcx97/_remote_module_non_sriptable.py 2022-03-17T19:49:32.9975158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:49:33.0300356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:49:33.0403258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:49:33.0778821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:49:46.4743540Z ok (16.249s) 2022-03-17T19:49:46.4747114Z 2022-03-17T19:49:46.4748168Z ---------------------------------------------------------------------- 2022-03-17T19:49:46.4748786Z Ran 1 test in 16.249s 2022-03-17T19:49:46.4749087Z 2022-03-17T19:49:46.4749291Z OK 2022-03-17T19:49:46.4749518Z 2022-03-17T19:49:46.4749757Z Generating XML reports... 2022-03-17T19:49:46.4797184Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194930.xml 2022-03-17T19:49:47.5267450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13whg_xk 2022-03-17T19:49:47.5268774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13whg_xk/_remote_module_non_sriptable.py 2022-03-17T19:49:47.8788269Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:49:47.8803229Z 2022-03-17T19:49:47.8803502Z Running tests... 2022-03-17T19:49:47.8803952Z ---------------------------------------------------------------------- 2022-03-17T19:49:49.3983260Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:49:49.4405386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73599 2022-03-17T19:49:49.4527437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73600 2022-03-17T19:49:49.4651711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73601 2022-03-17T19:49:49.4777796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73602 2022-03-17T19:49:50.3410416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcrrwx9r 2022-03-17T19:49:50.3411023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcrrwx9r/_remote_module_non_sriptable.py 2022-03-17T19:49:50.3415835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6te1at0c 2022-03-17T19:49:50.3418682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6te1at0c/_remote_module_non_sriptable.py 2022-03-17T19:49:50.3419397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2eou2wy7 2022-03-17T19:49:50.3423048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2eou2wy7/_remote_module_non_sriptable.py 2022-03-17T19:49:50.3435576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexf9j9e5 2022-03-17T19:49:50.3438362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexf9j9e5/_remote_module_non_sriptable.py 2022-03-17T19:49:50.6880047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:49:50.6895011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:49:50.6913090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:49:50.6961073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:06.0085267Z ok (18.128s) 2022-03-17T19:50:06.0086280Z 2022-03-17T19:50:06.0086711Z ---------------------------------------------------------------------- 2022-03-17T19:50:06.0087078Z Ran 1 test in 18.128s 2022-03-17T19:50:06.0087244Z 2022-03-17T19:50:06.0087327Z OK 2022-03-17T19:50:06.0087486Z 2022-03-17T19:50:06.0087627Z Generating XML reports... 2022-03-17T19:50:06.0132521Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194947.xml 2022-03-17T19:50:07.0541662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihs0oyc4 2022-03-17T19:50:07.0542565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihs0oyc4/_remote_module_non_sriptable.py 2022-03-17T19:50:07.4077935Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:50:07.4092551Z 2022-03-17T19:50:07.4092817Z Running tests... 2022-03-17T19:50:07.4093265Z ---------------------------------------------------------------------- 2022-03-17T19:50:08.9265446Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:50:08.9686275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73958 2022-03-17T19:50:08.9808800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73959 2022-03-17T19:50:08.9934942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73960 2022-03-17T19:50:09.0062073Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73961 2022-03-17T19:50:09.7810540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpguta_dvo 2022-03-17T19:50:09.7812025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpguta_dvo/_remote_module_non_sriptable.py 2022-03-17T19:50:09.7875238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cg1jty2 2022-03-17T19:50:09.7878130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cg1jty2/_remote_module_non_sriptable.py 2022-03-17T19:50:09.8013462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_nnuenj 2022-03-17T19:50:09.8016099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_nnuenj/_remote_module_non_sriptable.py 2022-03-17T19:50:09.8403077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd01zkcm5 2022-03-17T19:50:09.8405405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd01zkcm5/_remote_module_non_sriptable.py 2022-03-17T19:50:10.1302266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:10.1370212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:10.1449718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:50:10.1902531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:50:23.5327365Z ok (16.123s) 2022-03-17T19:50:23.5327588Z 2022-03-17T19:50:23.5327999Z ---------------------------------------------------------------------- 2022-03-17T19:50:23.5328362Z Ran 1 test in 16.123s 2022-03-17T19:50:23.5328511Z 2022-03-17T19:50:23.5328602Z OK 2022-03-17T19:50:23.5328737Z 2022-03-17T19:50:23.5328872Z Generating XML reports... 2022-03-17T19:50:23.5373980Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317195007.xml 2022-03-17T19:50:24.5793486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmyuxqzw 2022-03-17T19:50:24.5794365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmyuxqzw/_remote_module_non_sriptable.py 2022-03-17T19:50:24.9354613Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:50:24.9369744Z 2022-03-17T19:50:24.9370041Z Running tests... 2022-03-17T19:50:24.9370490Z ---------------------------------------------------------------------- 2022-03-17T19:50:26.4930277Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:50:26.5350663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74311 2022-03-17T19:50:26.5474572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74312 2022-03-17T19:50:26.5598117Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74313 2022-03-17T19:50:26.5725197Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74314 2022-03-17T19:50:27.3469401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ma9aadh 2022-03-17T19:50:27.3470522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ma9aadh/_remote_module_non_sriptable.py 2022-03-17T19:50:27.3727418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_l9osvw0 2022-03-17T19:50:27.3730277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_l9osvw0/_remote_module_non_sriptable.py 2022-03-17T19:50:27.4029581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5x990h2r 2022-03-17T19:50:27.4032561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5x990h2r/_remote_module_non_sriptable.py 2022-03-17T19:50:27.4080463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsli3e9b 2022-03-17T19:50:27.4084020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsli3e9b/_remote_module_non_sriptable.py 2022-03-17T19:50:27.6992938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:27.7135047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:27.7464211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:50:27.7497671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:50:43.4039228Z ok (18.467s) 2022-03-17T19:50:43.4039743Z 2022-03-17T19:50:43.4040179Z ---------------------------------------------------------------------- 2022-03-17T19:50:43.4040511Z Ran 1 test in 18.467s 2022-03-17T19:50:43.4040676Z 2022-03-17T19:50:43.4040773Z OK 2022-03-17T19:50:43.4040912Z 2022-03-17T19:50:43.4041063Z Generating XML reports... 2022-03-17T19:50:43.4085049Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317195024.xml 2022-03-17T19:50:44.4369110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2kjcmi9c 2022-03-17T19:50:44.4370067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2kjcmi9c/_remote_module_non_sriptable.py 2022-03-17T19:50:44.7924486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:50:44.7939227Z 2022-03-17T19:50:44.7939660Z Running tests... 2022-03-17T19:50:44.7940181Z ---------------------------------------------------------------------- 2022-03-17T19:50:46.3160143Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:50:46.3577064Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74670 2022-03-17T19:50:46.3699375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74671 2022-03-17T19:50:46.3824893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74672 2022-03-17T19:50:46.3950247Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74673 2022-03-17T19:50:47.1821070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xksj58k 2022-03-17T19:50:47.1821690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xksj58k/_remote_module_non_sriptable.py 2022-03-17T19:50:47.2024080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiq8dl6rm 2022-03-17T19:50:47.2026339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiq8dl6rm/_remote_module_non_sriptable.py 2022-03-17T19:50:47.2318775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp512f4nrb 2022-03-17T19:50:47.2321075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp512f4nrb/_remote_module_non_sriptable.py 2022-03-17T19:50:47.2617054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_j_1zztu 2022-03-17T19:50:47.2619535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_j_1zztu/_remote_module_non_sriptable.py 2022-03-17T19:50:47.5251029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:47.5458424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:50:47.5840096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:47.6072296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:50:50.9055694Z ok (6.111s) 2022-03-17T19:50:50.9055907Z 2022-03-17T19:50:50.9056311Z ---------------------------------------------------------------------- 2022-03-17T19:50:50.9056647Z Ran 1 test in 6.112s 2022-03-17T19:50:50.9057092Z 2022-03-17T19:50:50.9057213Z OK 2022-03-17T19:50:50.9057261Z 2022-03-17T19:50:50.9057396Z Generating XML reports... 2022-03-17T19:50:50.9098819Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317195044.xml 2022-03-17T19:50:51.9528402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphz08mdwd 2022-03-17T19:50:51.9530155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphz08mdwd/_remote_module_non_sriptable.py 2022-03-17T19:50:52.3060793Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:50:52.3076886Z 2022-03-17T19:50:52.3077343Z Running tests... 2022-03-17T19:50:52.3077815Z ---------------------------------------------------------------------- 2022-03-17T19:50:53.8458325Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:50:53.8876902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75023 2022-03-17T19:50:53.8998967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75024 2022-03-17T19:50:53.9124638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75025 2022-03-17T19:50:53.9254033Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75026 2022-03-17T19:50:54.7318504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55nigloz 2022-03-17T19:50:54.7319702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55nigloz/_remote_module_non_sriptable.py 2022-03-17T19:50:54.7514480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrp0lvof 2022-03-17T19:50:54.7517561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrp0lvof/_remote_module_non_sriptable.py 2022-03-17T19:50:54.7956377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8s73w85e 2022-03-17T19:50:54.7958913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8s73w85e/_remote_module_non_sriptable.py 2022-03-17T19:50:54.8074185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec7zk330 2022-03-17T19:50:54.8076712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec7zk330/_remote_module_non_sriptable.py 2022-03-17T19:50:55.0827495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:50:55.0998188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:50:55.1373441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:50:55.1505628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:51:01.1403172Z ok (8.832s) 2022-03-17T19:51:01.1403427Z 2022-03-17T19:51:01.1403827Z ---------------------------------------------------------------------- 2022-03-17T19:51:01.1404166Z Ran 1 test in 8.833s 2022-03-17T19:51:01.1404330Z 2022-03-17T19:51:01.1404427Z OK 2022-03-17T19:51:01.1404559Z 2022-03-17T19:51:01.1404662Z Generating XML reports... 2022-03-17T19:51:01.1447231Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317195052.xml 2022-03-17T19:51:02.2016961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvymgpie_ 2022-03-17T19:51:02.2017584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvymgpie_/_remote_module_non_sriptable.py 2022-03-17T19:51:02.5595442Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:51:02.5609114Z 2022-03-17T19:51:02.5609339Z Running tests... 2022-03-17T19:51:02.5609788Z ---------------------------------------------------------------------- 2022-03-17T19:51:04.1014751Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:04.1446683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75676 2022-03-17T19:51:04.1571389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75677 2022-03-17T19:51:04.1700750Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75678 2022-03-17T19:51:04.1827334Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75679 2022-03-17T19:51:04.9828452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxdkfghx 2022-03-17T19:51:04.9829497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxelvzxst 2022-03-17T19:51:04.9830064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxdkfghx/_remote_module_non_sriptable.py 2022-03-17T19:51:04.9830589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxelvzxst/_remote_module_non_sriptable.py 2022-03-17T19:51:05.0204439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfeeos11z 2022-03-17T19:51:05.0205855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfeeos11z/_remote_module_non_sriptable.py 2022-03-17T19:51:05.0529935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59487j3o 2022-03-17T19:51:05.0532600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59487j3o/_remote_module_non_sriptable.py 2022-03-17T19:51:05.3290302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:05.3369621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:05.3623944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:51:05.3972786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:51:05.5874998Z skip (3.026s) 2022-03-17T19:51:05.5875360Z 2022-03-17T19:51:05.5876114Z ---------------------------------------------------------------------- 2022-03-17T19:51:05.5876620Z Ran 1 test in 3.026s 2022-03-17T19:51:05.5876788Z 2022-03-17T19:51:05.5876896Z OK (skipped=1) 2022-03-17T19:51:05.5877051Z 2022-03-17T19:51:05.5877179Z Generating XML reports... 2022-03-17T19:51:05.5920638Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220317195102.xml 2022-03-17T19:51:06.6354811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpov2ii634 2022-03-17T19:51:06.6355684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpov2ii634/_remote_module_non_sriptable.py 2022-03-17T19:51:06.9868134Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:51:06.9883450Z 2022-03-17T19:51:06.9883708Z Running tests... 2022-03-17T19:51:06.9884165Z ---------------------------------------------------------------------- 2022-03-17T19:51:08.5095370Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:08.5515017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75857 2022-03-17T19:51:08.5637269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75858 2022-03-17T19:51:08.5762167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75859 2022-03-17T19:51:08.5889590Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75860 2022-03-17T19:51:09.4040586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0m0ycv5g 2022-03-17T19:51:09.4041678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0m0ycv5g/_remote_module_non_sriptable.py 2022-03-17T19:51:09.4045306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pskct1l 2022-03-17T19:51:09.4048247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pskct1l/_remote_module_non_sriptable.py 2022-03-17T19:51:09.4048772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4r33rzql 2022-03-17T19:51:09.4051311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4r33rzql/_remote_module_non_sriptable.py 2022-03-17T19:51:09.4399800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugh32b2_ 2022-03-17T19:51:09.4402378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugh32b2_/_remote_module_non_sriptable.py 2022-03-17T19:51:09.7470599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:51:09.7511927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:09.7530940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:09.7892615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:51:09.9936775Z skip (3.005s) 2022-03-17T19:51:09.9936997Z 2022-03-17T19:51:09.9937419Z ---------------------------------------------------------------------- 2022-03-17T19:51:09.9937760Z Ran 1 test in 3.005s 2022-03-17T19:51:09.9937903Z 2022-03-17T19:51:09.9938020Z OK (skipped=1) 2022-03-17T19:51:09.9938167Z 2022-03-17T19:51:09.9938283Z Generating XML reports... 2022-03-17T19:51:09.9983066Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220317195106.xml 2022-03-17T19:51:11.0372112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpux0zzd82 2022-03-17T19:51:11.0372864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpux0zzd82/_remote_module_non_sriptable.py 2022-03-17T19:51:11.3892235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent 2022-03-17T19:51:11.3908905Z 2022-03-17T19:51:11.3909078Z Running tests... 2022-03-17T19:51:11.3909544Z ---------------------------------------------------------------------- 2022-03-17T19:51:12.9220164Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:12.9639897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76038 2022-03-17T19:51:12.9763060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76039 2022-03-17T19:51:12.9886844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76040 2022-03-17T19:51:13.0014302Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76041 2022-03-17T19:51:13.8202350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2r8rp0a 2022-03-17T19:51:13.8203872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2r8rp0a/_remote_module_non_sriptable.py 2022-03-17T19:51:13.8277870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy_s7m6e 2022-03-17T19:51:13.8280714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy_s7m6e/_remote_module_non_sriptable.py 2022-03-17T19:51:13.8404799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp75emhoc 2022-03-17T19:51:13.8407568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp75emhoc/_remote_module_non_sriptable.py 2022-03-17T19:51:13.8494377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj30_vx5k 2022-03-17T19:51:13.8497143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj30_vx5k/_remote_module_non_sriptable.py 2022-03-17T19:51:14.1676086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:14.1716219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:51:14.1847511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:14.1956756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:51:14.4061263Z skip (3.015s) 2022-03-17T19:51:14.4061864Z 2022-03-17T19:51:14.4062302Z ---------------------------------------------------------------------- 2022-03-17T19:51:14.4062641Z Ran 1 test in 3.015s 2022-03-17T19:51:14.4062792Z 2022-03-17T19:51:14.4062876Z OK (skipped=1) 2022-03-17T19:51:14.4062998Z 2022-03-17T19:51:14.4063136Z Generating XML reports... 2022-03-17T19:51:14.4109671Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220317195111.xml 2022-03-17T19:51:14.7790869Z Running distributed/rpc/test_faulty_agent ... [2022-03-17 19:51:14.778547] 2022-03-17T19:51:14.7791695Z Executing ['/opt/conda/bin/python', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:51:14.778658] 2022-03-17T19:51:15.5755670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzujzqkf8 2022-03-17T19:51:15.5756970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzujzqkf8/_remote_module_non_sriptable.py 2022-03-17T19:51:16.0218621Z Running distributed/rpc/test_tensorpipe_agent ... [2022-03-17 19:51:16.021367] 2022-03-17T19:51:16.0219251Z Executing ['/opt/conda/bin/python', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:51:16.021471] 2022-03-17T19:51:16.7622290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9i2hzrz 2022-03-17T19:51:16.7623675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9i2hzrz/_remote_module_non_sriptable.py 2022-03-17T19:51:17.2115039Z Running distributed/test_c10d_common ... [2022-03-17 19:51:17.210998] 2022-03-17T19:51:17.2115761Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:51:17.211105] 2022-03-17T19:51:18.0147356Z test_debug_level (__main__.CommTest) 2022-03-17T19:51:18.0148058Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-03-17T19:51:18.0148535Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-03-17T19:51:18.0148956Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) 2022-03-17T19:51:18.0149359Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) 2022-03-17T19:51:18.0149815Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) 2022-03-17T19:51:18.0150244Z test_collectives (__main__.PythonProcessGroupExtensionTest) 2022-03-17T19:51:18.0150699Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) 2022-03-17T19:51:18.0151059Z test_send_recv (__main__.PythonProcessGroupExtensionTest) 2022-03-17T19:51:18.8075238Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:18.8091263Z 2022-03-17T19:51:18.8091422Z Running tests... 2022-03-17T19:51:18.8092235Z ---------------------------------------------------------------------- 2022-03-17T19:51:20.3379879Z test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:20.3798362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76324 2022-03-17T19:51:20.3920294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76325 2022-03-17T19:51:21.1960979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:21.2244934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:21.3960356Z ok (2.587s) 2022-03-17T19:51:21.3960547Z 2022-03-17T19:51:21.3960994Z ---------------------------------------------------------------------- 2022-03-17T19:51:21.3961679Z Ran 1 test in 2.587s 2022-03-17T19:51:21.3961852Z 2022-03-17T19:51:21.3962002Z OK 2022-03-17T19:51:21.3962126Z 2022-03-17T19:51:21.3962208Z Generating XML reports... 2022-03-17T19:51:21.4020303Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220317195118.xml 2022-03-17T19:51:22.4330527Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:22.4347021Z 2022-03-17T19:51:22.4347166Z Running tests... 2022-03-17T19:51:22.4347872Z ---------------------------------------------------------------------- 2022-03-17T19:51:23.9760917Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:23.9920459Z ok (1.557s) 2022-03-17T19:51:23.9920906Z 2022-03-17T19:51:23.9921343Z ---------------------------------------------------------------------- 2022-03-17T19:51:23.9921662Z Ran 1 test in 1.557s 2022-03-17T19:51:23.9921841Z 2022-03-17T19:51:23.9921936Z OK 2022-03-17T19:51:23.9922072Z 2022-03-17T19:51:23.9922198Z Generating XML reports... 2022-03-17T19:51:23.9962914Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317195122.xml 2022-03-17T19:51:24.9942901Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:24.9958054Z 2022-03-17T19:51:24.9958329Z Running tests... 2022-03-17T19:51:24.9959202Z ---------------------------------------------------------------------- 2022-03-17T19:51:26.5301587Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:26.5458025Z ok (1.550s) 2022-03-17T19:51:26.5458305Z 2022-03-17T19:51:26.5459041Z ---------------------------------------------------------------------- 2022-03-17T19:51:26.5459603Z Ran 1 test in 1.550s 2022-03-17T19:51:26.5459921Z 2022-03-17T19:51:26.5460079Z OK 2022-03-17T19:51:26.5460726Z 2022-03-17T19:51:26.5460980Z Generating XML reports... 2022-03-17T19:51:26.5502301Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317195124.xml 2022-03-17T19:51:27.5514506Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:27.5529008Z 2022-03-17T19:51:27.5530642Z Running tests... 2022-03-17T19:51:27.5531413Z ---------------------------------------------------------------------- 2022-03-17T19:51:29.0943186Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:29.1098215Z ok (1.557s) 2022-03-17T19:51:29.1098392Z 2022-03-17T19:51:29.1099230Z ---------------------------------------------------------------------- 2022-03-17T19:51:29.1099767Z Ran 1 test in 1.557s 2022-03-17T19:51:29.1099938Z 2022-03-17T19:51:29.1099971Z OK 2022-03-17T19:51:29.1100101Z 2022-03-17T19:51:29.1100579Z Generating XML reports... 2022-03-17T19:51:29.1140413Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317195127.xml 2022-03-17T19:51:30.1157867Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:30.1172350Z 2022-03-17T19:51:30.1172496Z Running tests... 2022-03-17T19:51:30.1173242Z ---------------------------------------------------------------------- 2022-03-17T19:51:31.6514108Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:31.6669375Z ok (1.550s) 2022-03-17T19:51:31.6670304Z 2022-03-17T19:51:31.6670843Z ---------------------------------------------------------------------- 2022-03-17T19:51:31.6671215Z Ran 1 test in 1.550s 2022-03-17T19:51:31.6671377Z 2022-03-17T19:51:31.6671471Z OK 2022-03-17T19:51:31.6671612Z 2022-03-17T19:51:31.6671742Z Generating XML reports... 2022-03-17T19:51:31.6711697Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317195130.xml 2022-03-17T19:51:32.6832732Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:32.6847624Z 2022-03-17T19:51:32.6847767Z Running tests... 2022-03-17T19:51:32.6848809Z ---------------------------------------------------------------------- 2022-03-17T19:51:34.2101444Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:34.2516991Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76577 2022-03-17T19:51:34.2638814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76578 2022-03-17T19:51:34.2760823Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76579 2022-03-17T19:51:34.2885250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76580 2022-03-17T19:51:35.0896794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:35.1002710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:35.1299613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:51:35.1619782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:51:35.2926037Z ok (2.607s) 2022-03-17T19:51:35.2926286Z 2022-03-17T19:51:35.2927074Z ---------------------------------------------------------------------- 2022-03-17T19:51:35.2927669Z Ran 1 test in 2.608s 2022-03-17T19:51:35.2927836Z 2022-03-17T19:51:35.2927903Z OK 2022-03-17T19:51:35.2928065Z 2022-03-17T19:51:35.2928200Z Generating XML reports... 2022-03-17T19:51:35.2984232Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317195132.xml 2022-03-17T19:51:36.3195894Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:36.3210510Z 2022-03-17T19:51:36.3210908Z Running tests... 2022-03-17T19:51:36.3211404Z ---------------------------------------------------------------------- 2022-03-17T19:51:37.8464008Z test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:37.8882744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76758 2022-03-17T19:51:37.9005873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76759 2022-03-17T19:51:37.9131235Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76760 2022-03-17T19:51:37.9257949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76761 2022-03-17T19:51:38.7145939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:51:38.7157839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:38.7421454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:51:38.7431665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:51:38.7690189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:38.7700604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:51:39.7158035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:51:39.7171107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:51:39.7173218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:51:39.7202515Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:51:39.7261355Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:51:39.7266702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:51:41.6344300Z ok (5.313s) 2022-03-17T19:51:41.6344753Z 2022-03-17T19:51:41.6345266Z ---------------------------------------------------------------------- 2022-03-17T19:51:41.6345721Z Ran 1 test in 5.313s 2022-03-17T19:51:41.6345898Z 2022-03-17T19:51:41.6345974Z OK 2022-03-17T19:51:41.6346112Z 2022-03-17T19:51:41.6346493Z Generating XML reports... 2022-03-17T19:51:41.6410997Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317195136.xml 2022-03-17T19:51:42.6847319Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:42.6861942Z 2022-03-17T19:51:42.6862550Z Running tests... 2022-03-17T19:51:42.6863103Z ---------------------------------------------------------------------- 2022-03-17T19:51:44.2187403Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:44.2608334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76948 2022-03-17T19:51:44.2729043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76949 2022-03-17T19:51:44.2854239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76950 2022-03-17T19:51:44.2983078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76951 2022-03-17T19:51:45.0723597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:51:45.0802729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:51:45.1421128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:45.1457926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:45.3023132Z ok (2.616s) 2022-03-17T19:51:45.3023541Z 2022-03-17T19:51:45.3024221Z ---------------------------------------------------------------------- 2022-03-17T19:51:45.3024896Z Ran 1 test in 2.616s 2022-03-17T19:51:45.3025240Z 2022-03-17T19:51:45.3025433Z OK 2022-03-17T19:51:45.3025651Z 2022-03-17T19:51:45.3025828Z Generating XML reports... 2022-03-17T19:51:45.3082436Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317195142.xml 2022-03-17T19:51:46.3345576Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common 2022-03-17T19:51:46.3360360Z 2022-03-17T19:51:46.3360800Z Running tests... 2022-03-17T19:51:46.3361357Z ---------------------------------------------------------------------- 2022-03-17T19:51:47.8781239Z test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:47.9204154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77129 2022-03-17T19:51:47.9325719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77130 2022-03-17T19:51:47.9453499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77131 2022-03-17T19:51:47.9580985Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77132 2022-03-17T19:51:48.8124885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:48.8229777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:48.8240269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:51:48.8391658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:51:48.8401661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:51:48.8509094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:51:48.8519323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:51:48.8544971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:51:48.8546691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:51:48.8547531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:51:48.8606414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:51:48.8623114Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:51:50.6653466Z ok (4.329s) 2022-03-17T19:51:50.6653845Z 2022-03-17T19:51:50.6654382Z ---------------------------------------------------------------------- 2022-03-17T19:51:50.6654732Z Ran 1 test in 4.329s 2022-03-17T19:51:50.6654899Z 2022-03-17T19:51:50.6655007Z OK 2022-03-17T19:51:50.6655109Z 2022-03-17T19:51:50.6655255Z Generating XML reports... 2022-03-17T19:51:50.6710416Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317195146.xml 2022-03-17T19:51:51.0217811Z Running distributed/test_c10d_gloo ... [2022-03-17 19:51:51.021217] 2022-03-17T19:51:51.0218575Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:51:51.021327] 2022-03-17T19:51:51.8525309Z , <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>]> 2022-03-17T19:51:51.8526352Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) 2022-03-17T19:51:51.8526707Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) 2022-03-17T19:51:51.8527073Z test_gloo_barrier_device_ids (__main__.CommTest) 2022-03-17T19:51:51.8527394Z test_gloo_warn_not_in_group (__main__.CommTest) 2022-03-17T19:51:51.8527702Z test_sequence_num_incremented_gloo_default (__main__.CommTest) 2022-03-17T19:51:51.8528113Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) 2022-03-17T19:51:51.8528507Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) 2022-03-17T19:51:51.8528900Z test_sequence_num_set_gloo_new_group (__main__.CommTest) 2022-03-17T19:51:51.8534199Z , <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>]> 2022-03-17T19:51:51.8542822Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8543314Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8543788Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8544324Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8544812Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8545316Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8545812Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8546290Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8546743Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8547230Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8547733Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8548216Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8548716Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8549317Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8549767Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8550213Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8550631Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8551062Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8551499Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8552026Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8552504Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8552948Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8553434Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8553902Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8554392Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8554850Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8555249Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8555665Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8556087Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8556508Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8556938Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8557365Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8557772Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8558161Z test_sparse_gradients (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8558578Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:51:51.8558955Z 2022-03-17T19:51:51.8563858Z , <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]> 2022-03-17T19:51:51.8568819Z test_allgather_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8569199Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8569571Z test_allgather_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8569926Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8570323Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8570681Z test_allgather_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8571046Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8571411Z test_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8571756Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8572155Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8572554Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8572940Z test_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8573298Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8573687Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8574091Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8574464Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8574864Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8575237Z test_allreduce_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8575586Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8575962Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8576329Z test_broadcast_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8576673Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8577018Z test_broadcast_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8577380Z test_broadcast_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8577746Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8578089Z test_empty_tensors (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8578441Z test_gather_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8578853Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8579211Z test_gather_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8579538Z test_gather_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8579892Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8580701Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8581072Z test_reduce_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8581430Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8581768Z test_reduce_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8582200Z test_reduce_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8582568Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8582906Z test_round_robin (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8583272Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8583628Z test_scatter_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8583992Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8584332Z test_scatter_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8584696Z test_scatter_stress (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8585048Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8585398Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8585776Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8586164Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8586545Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) 2022-03-17T19:51:51.8587391Z , <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]> 2022-03-17T19:51:51.8588174Z test_forward_backward (__main__.ReducerTest) 2022-03-17T19:51:51.8588512Z test_forward_backward_optimizer (__main__.ReducerTest) 2022-03-17T19:51:51.8588875Z test_forward_backward_unused_parameters (__main__.ReducerTest) 2022-03-17T19:51:51.8589212Z test_multi_dtype_multi_bucket (__main__.ReducerTest) 2022-03-17T19:51:51.8589552Z test_multi_dtype_single_bucket (__main__.ReducerTest) 2022-03-17T19:51:51.8589876Z test_single_dtype_single_bucket (__main__.ReducerTest) 2022-03-17T19:51:51.8590290Z ]> 2022-03-17T19:51:51.8590694Z test_logging_init (__main__.RendezvousEnvTest) 2022-03-17T19:51:51.8590997Z 2022-03-17T19:51:51.8591403Z ]> 2022-03-17T19:51:51.8591805Z test_default_store_timeout_gloo (__main__.TimeoutTest) 2022-03-17T19:51:52.6576247Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:51:52.6591217Z 2022-03-17T19:51:52.6591725Z Running tests... 2022-03-17T19:51:52.6592217Z ---------------------------------------------------------------------- 2022-03-17T19:51:54.2103372Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:54.2529470Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77354 2022-03-17T19:51:54.2651821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77355 2022-03-17T19:51:55.1033264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:55.1291871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:51:55.3690560Z ok (2.710s) 2022-03-17T19:51:55.3690846Z 2022-03-17T19:51:55.3691209Z ---------------------------------------------------------------------- 2022-03-17T19:51:55.3691558Z Ran 1 test in 2.710s 2022-03-17T19:51:55.3691724Z 2022-03-17T19:51:55.3691814Z OK 2022-03-17T19:51:55.3691941Z 2022-03-17T19:51:55.3692033Z Generating XML reports... 2022-03-17T19:51:55.3734970Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195152.xml 2022-03-17T19:51:56.4097766Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:51:56.4111507Z 2022-03-17T19:51:56.4111731Z Running tests... 2022-03-17T19:51:56.4112411Z ---------------------------------------------------------------------- 2022-03-17T19:51:57.9338378Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:51:57.9764665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77469 2022-03-17T19:51:57.9885890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77470 2022-03-17T19:51:58.8437933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:51:58.8449142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:00.4947301Z ok (4.083s) 2022-03-17T19:52:00.4947517Z 2022-03-17T19:52:00.4947917Z ---------------------------------------------------------------------- 2022-03-17T19:52:00.4948247Z Ran 1 test in 4.084s 2022-03-17T19:52:00.4948417Z 2022-03-17T19:52:00.4948517Z OK 2022-03-17T19:52:00.4948656Z 2022-03-17T19:52:00.4948791Z Generating XML reports... 2022-03-17T19:52:00.4991686Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195156.xml 2022-03-17T19:52:01.5579086Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:01.5593342Z 2022-03-17T19:52:01.5593642Z Running tests... 2022-03-17T19:52:01.5594339Z ---------------------------------------------------------------------- 2022-03-17T19:52:03.1052038Z test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:03.1481301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77586 2022-03-17T19:52:03.1602196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77587 2022-03-17T19:52:04.0011113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:04.0350290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:04.0460606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:52:04.0462026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:52:04.0462921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:04.0463607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:04.2640962Z ok (2.704s) 2022-03-17T19:52:04.2641362Z 2022-03-17T19:52:04.2642097Z ---------------------------------------------------------------------- 2022-03-17T19:52:04.2642739Z Ran 1 test in 2.705s 2022-03-17T19:52:04.2643011Z 2022-03-17T19:52:04.2643165Z OK 2022-03-17T19:52:04.2643288Z 2022-03-17T19:52:04.2643428Z Generating XML reports... 2022-03-17T19:52:04.2685213Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195201.xml 2022-03-17T19:52:05.3076830Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:05.3090025Z 2022-03-17T19:52:05.3090434Z Running tests... 2022-03-17T19:52:05.3091370Z ---------------------------------------------------------------------- 2022-03-17T19:52:06.8435772Z test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:06.8865488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77701 2022-03-17T19:52:06.8986560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77702 2022-03-17T19:52:07.6990629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:07.7293891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:07.7502587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:52:07.7503124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:52:07.7503890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:07.7504875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:07.7505701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:52:07.7509144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:52:07.7509835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:52:07.7608899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:52:09.3047528Z ok (3.995s) 2022-03-17T19:52:09.3047730Z 2022-03-17T19:52:09.3048130Z ---------------------------------------------------------------------- 2022-03-17T19:52:09.3048487Z Ran 1 test in 3.996s 2022-03-17T19:52:09.3048636Z 2022-03-17T19:52:09.3048726Z OK 2022-03-17T19:52:09.3048869Z 2022-03-17T19:52:09.3049002Z Generating XML reports... 2022-03-17T19:52:09.3091617Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195205.xml 2022-03-17T19:52:10.3457889Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:10.3472146Z 2022-03-17T19:52:10.3472307Z Running tests... 2022-03-17T19:52:10.3472744Z ---------------------------------------------------------------------- 2022-03-17T19:52:11.8821523Z test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:11.9249133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77821 2022-03-17T19:52:11.9370327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77822 2022-03-17T19:52:12.7743848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:12.7781315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:12.7897780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:52:12.7898269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:52:12.7899024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:12.7899721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:12.8107319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:52:12.8107801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:52:12.8108440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:52:12.8109347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:52:14.4433452Z ok (4.096s) 2022-03-17T19:52:14.4433821Z 2022-03-17T19:52:14.4434460Z ---------------------------------------------------------------------- 2022-03-17T19:52:14.4435044Z Ran 1 test in 4.096s 2022-03-17T19:52:14.4435310Z 2022-03-17T19:52:14.4435479Z OK 2022-03-17T19:52:14.4435731Z 2022-03-17T19:52:14.4435967Z Generating XML reports... 2022-03-17T19:52:14.4481082Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195210.xml 2022-03-17T19:52:15.4956025Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:15.4968994Z 2022-03-17T19:52:15.4969312Z Running tests... 2022-03-17T19:52:15.4969741Z ---------------------------------------------------------------------- 2022-03-17T19:52:16.9985585Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:17.0406890Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77944 2022-03-17T19:52:17.0527051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77945 2022-03-17T19:52:17.8821461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:17.8963794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:18.0563694Z skip (2.559s) 2022-03-17T19:52:18.0563860Z 2022-03-17T19:52:18.0564290Z ---------------------------------------------------------------------- 2022-03-17T19:52:18.0564663Z Ran 1 test in 2.559s 2022-03-17T19:52:18.0564793Z 2022-03-17T19:52:18.0565241Z OK (skipped=1) 2022-03-17T19:52:18.0565448Z 2022-03-17T19:52:18.0565886Z Generating XML reports... 2022-03-17T19:52:18.0611039Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195215.xml 2022-03-17T19:52:19.0938911Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:19.0953207Z 2022-03-17T19:52:19.0953340Z Running tests... 2022-03-17T19:52:19.0953795Z ---------------------------------------------------------------------- 2022-03-17T19:52:20.6460592Z test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:20.6884421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78053 2022-03-17T19:52:20.7005666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78054 2022-03-17T19:52:21.5002582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:21.5041087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:21.5150189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:52:21.5150714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:52:21.5151403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:21.5152079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:21.7041319Z ok (2.608s) 2022-03-17T19:52:21.7041532Z 2022-03-17T19:52:21.7041898Z ---------------------------------------------------------------------- 2022-03-17T19:52:21.7042673Z Ran 1 test in 2.609s 2022-03-17T19:52:21.7043015Z 2022-03-17T19:52:21.7043114Z OK 2022-03-17T19:52:21.7043271Z 2022-03-17T19:52:21.7043406Z Generating XML reports... 2022-03-17T19:52:21.7084932Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195219.xml 2022-03-17T19:52:22.7475539Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:22.7489301Z 2022-03-17T19:52:22.7489416Z Running tests... 2022-03-17T19:52:22.7490311Z ---------------------------------------------------------------------- 2022-03-17T19:52:24.2877307Z test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:24.3303042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78168 2022-03-17T19:52:24.3426411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78169 2022-03-17T19:52:25.1462197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:25.1683083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:25.1874263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:52:25.1874774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:52:25.1875471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:25.1876177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:52:25.1982633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:52:25.1983120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:52:25.1983802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:52:25.1984422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:52:25.3462784Z ok (2.597s) 2022-03-17T19:52:25.3463346Z 2022-03-17T19:52:25.3463990Z ---------------------------------------------------------------------- 2022-03-17T19:52:25.3464777Z Ran 1 test in 2.597s 2022-03-17T19:52:25.3465089Z 2022-03-17T19:52:25.3465130Z OK 2022-03-17T19:52:25.3465266Z 2022-03-17T19:52:25.3465405Z Generating XML reports... 2022-03-17T19:52:25.3508513Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195222.xml 2022-03-17T19:52:26.3877867Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:26.3891521Z 2022-03-17T19:52:26.3891767Z Running tests... 2022-03-17T19:52:26.3892194Z ---------------------------------------------------------------------- 2022-03-17T19:52:26.3901088Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-03-17T19:52:27.9395117Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:27.9821964Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78289 2022-03-17T19:52:27.9944725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78290 2022-03-17T19:52:28.7979349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:28.8119310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:30.8013595Z ok (4.412s) 2022-03-17T19:52:30.8013842Z 2022-03-17T19:52:30.8014218Z ---------------------------------------------------------------------- 2022-03-17T19:52:30.8014492Z Ran 1 test in 4.412s 2022-03-17T19:52:30.8014660Z 2022-03-17T19:52:30.8014761Z OK 2022-03-17T19:52:30.8014879Z 2022-03-17T19:52:30.8015031Z Generating XML reports... 2022-03-17T19:52:30.8058053Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195226.xml 2022-03-17T19:52:31.8600902Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:31.8615042Z 2022-03-17T19:52:31.8615371Z Running tests... 2022-03-17T19:52:31.8615798Z ---------------------------------------------------------------------- 2022-03-17T19:52:31.8624968Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:52:33.4002878Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:33.4430717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78410 2022-03-17T19:52:33.4553138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78411 2022-03-17T19:52:34.2584840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:34.3026994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:36.2624760Z ok (4.401s) 2022-03-17T19:52:36.2625197Z 2022-03-17T19:52:36.2625820Z ---------------------------------------------------------------------- 2022-03-17T19:52:36.2626460Z Ran 1 test in 4.401s 2022-03-17T19:52:36.2626757Z 2022-03-17T19:52:36.2626823Z OK 2022-03-17T19:52:36.2627121Z 2022-03-17T19:52:36.2627417Z Generating XML reports... 2022-03-17T19:52:36.2671387Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195231.xml 2022-03-17T19:52:37.3266498Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:37.3280799Z 2022-03-17T19:52:37.3281306Z Running tests... 2022-03-17T19:52:37.3281746Z ---------------------------------------------------------------------- 2022-03-17T19:52:37.3292809Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:52:38.8556779Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:38.8986178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78531 2022-03-17T19:52:38.9109091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78532 2022-03-17T19:52:39.7424773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:39.7463211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:41.3904260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.3904704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.4212494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.4212984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.4366995Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:52:41.4367831Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:52:41.4368974Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:52:41.4369797Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:52:41.4484281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.4484742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.4701401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.4701909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.5001743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.5002264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.5265340Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.5265962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:41.8179993Z ok (4.490s) 2022-03-17T19:52:41.8180236Z 2022-03-17T19:52:41.8180859Z ---------------------------------------------------------------------- 2022-03-17T19:52:41.8181199Z Ran 1 test in 4.490s 2022-03-17T19:52:41.8181361Z 2022-03-17T19:52:41.8181472Z OK 2022-03-17T19:52:41.8182542Z 2022-03-17T19:52:41.8184454Z Generating XML reports... 2022-03-17T19:52:41.8226139Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195237.xml 2022-03-17T19:52:42.8703420Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:42.8716872Z 2022-03-17T19:52:42.8717210Z Running tests... 2022-03-17T19:52:42.8717650Z ---------------------------------------------------------------------- 2022-03-17T19:52:42.8728946Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:52:44.3832141Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:44.4254319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78652 2022-03-17T19:52:44.4374684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78653 2022-03-17T19:52:45.2497194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:45.2511208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:46.9173106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:46.9173613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:46.9537115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:46.9537611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:46.9717003Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:52:46.9717979Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:52:46.9719123Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:52:46.9719955Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:52:46.9850573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:46.9851111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:47.0102372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:47.0103173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:47.0461024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:47.0461523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:47.0765864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:47.0766362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:47.4443647Z ok (4.572s) 2022-03-17T19:52:47.4443987Z 2022-03-17T19:52:47.4444653Z ---------------------------------------------------------------------- 2022-03-17T19:52:47.4445026Z Ran 1 test in 4.573s 2022-03-17T19:52:47.4445159Z 2022-03-17T19:52:47.4445254Z OK 2022-03-17T19:52:47.4445387Z 2022-03-17T19:52:47.4445520Z Generating XML reports... 2022-03-17T19:52:47.4489394Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195242.xml 2022-03-17T19:52:48.4973357Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:48.4987373Z 2022-03-17T19:52:48.4987634Z Running tests... 2022-03-17T19:52:48.4988163Z ---------------------------------------------------------------------- 2022-03-17T19:52:48.4996840Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:52:50.0241851Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:50.0677545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78773 2022-03-17T19:52:50.0801518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78774 2022-03-17T19:52:50.9162794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:50.9327028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:52.5741258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:52.5741857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:52.6045257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:52.6045728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:52.8870252Z ok (4.388s) 2022-03-17T19:52:52.8870472Z 2022-03-17T19:52:52.8870860Z ---------------------------------------------------------------------- 2022-03-17T19:52:52.8871199Z Ran 1 test in 4.388s 2022-03-17T19:52:52.8871367Z 2022-03-17T19:52:52.8871456Z OK 2022-03-17T19:52:52.8871573Z 2022-03-17T19:52:52.8871702Z Generating XML reports... 2022-03-17T19:52:52.8915166Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195248.xml 2022-03-17T19:52:53.9553008Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:53.9567073Z 2022-03-17T19:52:53.9567320Z Running tests... 2022-03-17T19:52:53.9578933Z ---------------------------------------------------------------------- 2022-03-17T19:52:53.9579423Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:52:55.4978788Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:52:55.5408283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78894 2022-03-17T19:52:55.5530100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78895 2022-03-17T19:52:56.3636880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:52:56.3637364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:52:58.0249751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:58.0250243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:58.0582974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:58.0583417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:52:58.3598666Z ok (4.403s) 2022-03-17T19:52:58.3599098Z 2022-03-17T19:52:58.3599785Z ---------------------------------------------------------------------- 2022-03-17T19:52:58.3600452Z Ran 1 test in 4.403s 2022-03-17T19:52:58.3600634Z 2022-03-17T19:52:58.3600727Z OK 2022-03-17T19:52:58.3600864Z 2022-03-17T19:52:58.3600976Z Generating XML reports... 2022-03-17T19:52:58.3643990Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195253.xml 2022-03-17T19:52:59.4179733Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:52:59.4194244Z 2022-03-17T19:52:59.4194420Z Running tests... 2022-03-17T19:52:59.4195055Z ---------------------------------------------------------------------- 2022-03-17T19:52:59.4207084Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:53:00.9427143Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:00.9856952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79015 2022-03-17T19:53:00.9982424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79016 2022-03-17T19:53:01.8327745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:01.8707050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:03.5388607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:03.5389089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:03.5637203Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:53:03.5638743Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:53:03.6005418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:03.6006147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:04.0054015Z ok (4.586s) 2022-03-17T19:53:04.0054229Z 2022-03-17T19:53:04.0054674Z ---------------------------------------------------------------------- 2022-03-17T19:53:04.0055016Z Ran 1 test in 4.586s 2022-03-17T19:53:04.0055144Z 2022-03-17T19:53:04.0055242Z OK 2022-03-17T19:53:04.0055377Z 2022-03-17T19:53:04.0055507Z Generating XML reports... 2022-03-17T19:53:04.0098867Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195259.xml 2022-03-17T19:53:05.0511925Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:05.0525210Z 2022-03-17T19:53:05.0525597Z Running tests... 2022-03-17T19:53:05.0526249Z ---------------------------------------------------------------------- 2022-03-17T19:53:05.0538467Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:53:06.5779401Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:06.6216361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79136 2022-03-17T19:53:06.6341332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79137 2022-03-17T19:53:07.4547221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:07.5022497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:09.5411922Z ok (4.488s) 2022-03-17T19:53:09.5412130Z 2022-03-17T19:53:09.5412509Z ---------------------------------------------------------------------- 2022-03-17T19:53:09.5412835Z Ran 1 test in 4.489s 2022-03-17T19:53:09.5412986Z 2022-03-17T19:53:09.5413088Z OK 2022-03-17T19:53:09.5413195Z 2022-03-17T19:53:09.5413328Z Generating XML reports... 2022-03-17T19:53:09.5456237Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195305.xml 2022-03-17T19:53:10.6005600Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:10.6019568Z 2022-03-17T19:53:10.6019918Z Running tests... 2022-03-17T19:53:10.6020399Z ---------------------------------------------------------------------- 2022-03-17T19:53:10.6029199Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:53:12.1367560Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:12.1792044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79257 2022-03-17T19:53:12.1915064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79258 2022-03-17T19:53:12.9958233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:13.0496753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:14.7049044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:14.7049577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:14.7354847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:14.7355410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:15.0984781Z ok (4.496s) 2022-03-17T19:53:15.0985011Z 2022-03-17T19:53:15.0985449Z ---------------------------------------------------------------------- 2022-03-17T19:53:15.0985787Z Ran 1 test in 4.496s 2022-03-17T19:53:15.0985952Z 2022-03-17T19:53:15.0986007Z OK 2022-03-17T19:53:15.0986117Z 2022-03-17T19:53:15.0986249Z Generating XML reports... 2022-03-17T19:53:15.1029505Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195310.xml 2022-03-17T19:53:16.1659003Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:16.1675022Z 2022-03-17T19:53:16.1675506Z Running tests... 2022-03-17T19:53:16.1675989Z ---------------------------------------------------------------------- 2022-03-17T19:53:16.1689364Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:53:17.7170476Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:17.7600541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79378 2022-03-17T19:53:17.7725694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79379 2022-03-17T19:53:18.5805157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:18.5807023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:20.2158166Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:53:20.2166168Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:53:20.2446232Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:53:20.2447076Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:53:20.2448220Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:53:20.2449037Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:53:20.2559667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:20.2560155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:20.3091690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:20.3092193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:20.6795968Z ok (4.512s) 2022-03-17T19:53:20.6796165Z 2022-03-17T19:53:20.6796589Z ---------------------------------------------------------------------- 2022-03-17T19:53:20.6796881Z Ran 1 test in 4.512s 2022-03-17T19:53:20.6797053Z 2022-03-17T19:53:20.6797127Z OK 2022-03-17T19:53:20.6797260Z 2022-03-17T19:53:20.6797393Z Generating XML reports... 2022-03-17T19:53:20.6841137Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195316.xml 2022-03-17T19:53:21.7322559Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:21.7336089Z 2022-03-17T19:53:21.7336396Z Running tests... 2022-03-17T19:53:21.7336835Z ---------------------------------------------------------------------- 2022-03-17T19:53:21.7349568Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:53:23.2688828Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:23.3115793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79499 2022-03-17T19:53:23.3237820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79500 2022-03-17T19:53:24.1487911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:24.1681683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:25.8191889Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:53:25.8192813Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:53:25.8193945Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:53:25.8194757Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:53:25.8324194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:25.8324931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:25.8734900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:25.8735651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:26.2308331Z ok (4.497s) 2022-03-17T19:53:26.2308699Z 2022-03-17T19:53:26.2309372Z ---------------------------------------------------------------------- 2022-03-17T19:53:26.2309692Z Ran 1 test in 4.497s 2022-03-17T19:53:26.2309747Z 2022-03-17T19:53:26.2309847Z OK 2022-03-17T19:53:26.2309984Z 2022-03-17T19:53:26.2310097Z Generating XML reports... 2022-03-17T19:53:26.2353922Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195321.xml 2022-03-17T19:53:27.2966184Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:27.2979837Z 2022-03-17T19:53:27.2980320Z Running tests... 2022-03-17T19:53:27.2981143Z ---------------------------------------------------------------------- 2022-03-17T19:53:27.2996142Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:53:28.8517234Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:28.8946536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79620 2022-03-17T19:53:28.9074182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79621 2022-03-17T19:53:29.7037894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:29.7073420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:31.3411190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:31.3411756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:31.3763648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:31.3764102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:31.7141236Z ok (4.416s) 2022-03-17T19:53:31.7141461Z 2022-03-17T19:53:31.7141867Z ---------------------------------------------------------------------- 2022-03-17T19:53:31.7142156Z Ran 1 test in 4.416s 2022-03-17T19:53:31.7142319Z 2022-03-17T19:53:31.7142368Z OK 2022-03-17T19:53:31.7142503Z 2022-03-17T19:53:31.7142628Z Generating XML reports... 2022-03-17T19:53:31.7187356Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195327.xml 2022-03-17T19:53:32.7720361Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:32.7733884Z 2022-03-17T19:53:32.7734345Z Running tests... 2022-03-17T19:53:32.7735116Z ---------------------------------------------------------------------- 2022-03-17T19:53:32.7748846Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:53:34.3049581Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:34.3475606Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79741 2022-03-17T19:53:34.3598608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79742 2022-03-17T19:53:35.2325891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:35.2328903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:36.8731477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:36.8732041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:36.9038852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:36.9039378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:36.9252066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:36.9252565Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:36.9556923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:36.9557429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:53:37.2667006Z ok (4.493s) 2022-03-17T19:53:37.2667291Z 2022-03-17T19:53:37.2667721Z ---------------------------------------------------------------------- 2022-03-17T19:53:37.2668099Z Ran 1 test in 4.493s 2022-03-17T19:53:37.2668222Z 2022-03-17T19:53:37.2668317Z OK 2022-03-17T19:53:37.2668464Z 2022-03-17T19:53:37.2668588Z Generating XML reports... 2022-03-17T19:53:37.2712426Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195332.xml 2022-03-17T19:53:38.3286539Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:38.3300422Z 2022-03-17T19:53:38.3300730Z Running tests... 2022-03-17T19:53:38.3301432Z ---------------------------------------------------------------------- 2022-03-17T19:53:38.3311971Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest) 2022-03-17T19:53:39.8671529Z This unit test verifies whether the Future object is passed properly. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:39.9101055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79862 2022-03-17T19:53:39.9223566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79863 2022-03-17T19:53:40.7663030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:40.7724422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:40.9260995Z ok (2.596s) 2022-03-17T19:53:40.9261219Z 2022-03-17T19:53:40.9261933Z ---------------------------------------------------------------------- 2022-03-17T19:53:40.9262253Z Ran 1 test in 2.596s 2022-03-17T19:53:40.9262417Z 2022-03-17T19:53:40.9262489Z OK 2022-03-17T19:53:40.9262625Z 2022-03-17T19:53:40.9262756Z Generating XML reports... 2022-03-17T19:53:40.9305898Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195338.xml 2022-03-17T19:53:41.9748878Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:41.9764348Z 2022-03-17T19:53:41.9764993Z Running tests... 2022-03-17T19:53:41.9765470Z ---------------------------------------------------------------------- 2022-03-17T19:53:41.9775689Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest) 2022-03-17T19:53:43.5063467Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:43.5491577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79981 2022-03-17T19:53:43.5614036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79982 2022-03-17T19:53:44.3826086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:44.4259073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:46.0678380Z ok (4.091s) 2022-03-17T19:53:46.0678751Z 2022-03-17T19:53:46.0679383Z ---------------------------------------------------------------------- 2022-03-17T19:53:46.0679987Z Ran 1 test in 4.091s 2022-03-17T19:53:46.0680253Z 2022-03-17T19:53:46.0680418Z OK 2022-03-17T19:53:46.0680682Z 2022-03-17T19:53:46.0680923Z Generating XML reports... 2022-03-17T19:53:46.0724862Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195341.xml 2022-03-17T19:53:47.1325004Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:47.1340045Z 2022-03-17T19:53:47.1340487Z Running tests... 2022-03-17T19:53:47.1341248Z ---------------------------------------------------------------------- 2022-03-17T19:53:47.1353480Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest) 2022-03-17T19:53:48.6600312Z DDP communication hook can only be registered once. This test validates whether ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:48.7039526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80102 2022-03-17T19:53:48.7162745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80103 2022-03-17T19:53:49.5916930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:49.5948041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:49.8200592Z ok (2.686s) 2022-03-17T19:53:49.8200750Z 2022-03-17T19:53:49.8201180Z ---------------------------------------------------------------------- 2022-03-17T19:53:49.8201572Z Ran 1 test in 2.686s 2022-03-17T19:53:49.8201690Z 2022-03-17T19:53:49.8201776Z OK 2022-03-17T19:53:49.8201914Z 2022-03-17T19:53:49.8201987Z Generating XML reports... 2022-03-17T19:53:49.8247464Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195347.xml 2022-03-17T19:53:50.8681904Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:50.8696491Z 2022-03-17T19:53:50.8696719Z Running tests... 2022-03-17T19:53:50.8697182Z ---------------------------------------------------------------------- 2022-03-17T19:53:50.8714864Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest) 2022-03-17T19:53:52.3964983Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:52.4407772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80217 2022-03-17T19:53:52.4531887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80218 2022-03-17T19:53:53.2501345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:53.2959636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:53.4567034Z ok (2.587s) 2022-03-17T19:53:53.4567382Z 2022-03-17T19:53:53.4567802Z ---------------------------------------------------------------------- 2022-03-17T19:53:53.4568141Z Ran 1 test in 2.587s 2022-03-17T19:53:53.4568296Z 2022-03-17T19:53:53.4568385Z OK 2022-03-17T19:53:53.4568518Z 2022-03-17T19:53:53.4568911Z Generating XML reports... 2022-03-17T19:53:53.4619692Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195350.xml 2022-03-17T19:53:54.5031674Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:54.5045373Z 2022-03-17T19:53:54.5046123Z Running tests... 2022-03-17T19:53:54.5046614Z ---------------------------------------------------------------------- 2022-03-17T19:53:54.5057855Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest) 2022-03-17T19:53:56.0491114Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:56.0917678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80366 2022-03-17T19:53:56.1039810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80367 2022-03-17T19:53:56.8995253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:53:56.9545434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:53:57.1075262Z ok (2.603s) 2022-03-17T19:53:57.1075515Z 2022-03-17T19:53:57.1076091Z ---------------------------------------------------------------------- 2022-03-17T19:53:57.1076412Z Ran 1 test in 2.603s 2022-03-17T19:53:57.1076542Z 2022-03-17T19:53:57.1076614Z OK 2022-03-17T19:53:57.1076749Z 2022-03-17T19:53:57.1076868Z Generating XML reports... 2022-03-17T19:53:57.1120448Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195354.xml 2022-03-17T19:53:58.1544913Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:53:58.1559075Z 2022-03-17T19:53:58.1559299Z Running tests... 2022-03-17T19:53:58.1559745Z ---------------------------------------------------------------------- 2022-03-17T19:53:58.1575379Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest) 2022-03-17T19:53:59.6795101Z This test checks whether return annotation checked properly if defined. It also ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:53:59.7224108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80481 2022-03-17T19:53:59.7346570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80482 2022-03-17T19:54:00.5342478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:00.5419646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:00.7382624Z ok (2.582s) 2022-03-17T19:54:00.7382855Z 2022-03-17T19:54:00.7383246Z ---------------------------------------------------------------------- 2022-03-17T19:54:00.7383554Z Ran 1 test in 2.582s 2022-03-17T19:54:00.7383689Z 2022-03-17T19:54:00.7383779Z OK 2022-03-17T19:54:00.7383912Z 2022-03-17T19:54:00.7384056Z Generating XML reports... 2022-03-17T19:54:00.7427046Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195358.xml 2022-03-17T19:54:01.7803116Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:01.7816721Z 2022-03-17T19:54:01.7816972Z Running tests... 2022-03-17T19:54:01.7817400Z ---------------------------------------------------------------------- 2022-03-17T19:54:01.7838170Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest) 2022-03-17T19:54:03.3295844Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:03.3724457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80600 2022-03-17T19:54:03.3847567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80601 2022-03-17T19:54:04.1934371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:04.2416521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:04.2577409Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:54:05.8910786Z ok (4.109s) 2022-03-17T19:54:05.8911002Z 2022-03-17T19:54:05.8911629Z ---------------------------------------------------------------------- 2022-03-17T19:54:05.8911989Z Ran 1 test in 4.109s 2022-03-17T19:54:05.8912136Z 2022-03-17T19:54:05.8912226Z OK 2022-03-17T19:54:05.8912345Z 2022-03-17T19:54:05.8912478Z Generating XML reports... 2022-03-17T19:54:05.8956070Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195401.xml 2022-03-17T19:54:06.9523146Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:06.9538581Z 2022-03-17T19:54:06.9538891Z Running tests... 2022-03-17T19:54:06.9539332Z ---------------------------------------------------------------------- 2022-03-17T19:54:08.4924184Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:08.5355178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80721 2022-03-17T19:54:08.5478885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80722 2022-03-17T19:54:09.4206901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:09.4220014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:11.0550787Z ok (4.101s) 2022-03-17T19:54:11.0550923Z 2022-03-17T19:54:11.0551356Z ---------------------------------------------------------------------- 2022-03-17T19:54:11.0551710Z Ran 1 test in 4.101s 2022-03-17T19:54:11.0551872Z 2022-03-17T19:54:11.0554765Z OK 2022-03-17T19:54:11.0554909Z 2022-03-17T19:54:11.0555028Z Generating XML reports... 2022-03-17T19:54:11.0594761Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195406.xml 2022-03-17T19:54:12.1188440Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:12.1202278Z 2022-03-17T19:54:12.1202697Z Running tests... 2022-03-17T19:54:12.1203182Z ---------------------------------------------------------------------- 2022-03-17T19:54:13.6516574Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:13.6944773Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80842 2022-03-17T19:54:13.7068272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80843 2022-03-17T19:54:14.5136287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:14.5356528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:16.2132392Z ok (4.093s) 2022-03-17T19:54:16.2132652Z 2022-03-17T19:54:16.2133068Z ---------------------------------------------------------------------- 2022-03-17T19:54:16.2133404Z Ran 1 test in 4.093s 2022-03-17T19:54:16.2133559Z 2022-03-17T19:54:16.2133595Z OK 2022-03-17T19:54:16.2133685Z 2022-03-17T19:54:16.2133933Z Generating XML reports... 2022-03-17T19:54:16.2178871Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195412.xml 2022-03-17T19:54:17.2727421Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:17.2740906Z 2022-03-17T19:54:17.2741494Z Running tests... 2022-03-17T19:54:17.2742219Z ---------------------------------------------------------------------- 2022-03-17T19:54:18.8322296Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:18.8753616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80963 2022-03-17T19:54:18.8877865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80964 2022-03-17T19:54:19.7192677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:19.7561677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:19.7737383Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:54:19.7738519Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:54:19.7739903Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T19:54:19.7741230Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T19:54:21.3939212Z ok (4.119s) 2022-03-17T19:54:21.3939427Z 2022-03-17T19:54:21.3939903Z ---------------------------------------------------------------------- 2022-03-17T19:54:21.3940688Z Ran 1 test in 4.120s 2022-03-17T19:54:21.3940859Z 2022-03-17T19:54:21.3940956Z OK 2022-03-17T19:54:21.3941073Z 2022-03-17T19:54:21.3941296Z Generating XML reports... 2022-03-17T19:54:21.3984395Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195417.xml 2022-03-17T19:54:22.4534513Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:22.4548743Z 2022-03-17T19:54:22.4549014Z Running tests... 2022-03-17T19:54:22.4549493Z ---------------------------------------------------------------------- 2022-03-17T19:54:23.9806801Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:24.0232992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81084 2022-03-17T19:54:24.0355173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81085 2022-03-17T19:54:24.8915707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:24.9044001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:26.5550617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:26.5551583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:26.8422119Z ok (4.387s) 2022-03-17T19:54:26.8422344Z 2022-03-17T19:54:26.8422733Z ---------------------------------------------------------------------- 2022-03-17T19:54:26.8423054Z Ran 1 test in 4.387s 2022-03-17T19:54:26.8423219Z 2022-03-17T19:54:26.8423309Z OK 2022-03-17T19:54:26.8423441Z 2022-03-17T19:54:26.8423560Z Generating XML reports... 2022-03-17T19:54:26.8465896Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195422.xml 2022-03-17T19:54:27.9080027Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:27.9094254Z 2022-03-17T19:54:27.9094707Z Running tests... 2022-03-17T19:54:27.9095212Z ---------------------------------------------------------------------- 2022-03-17T19:54:29.4730071Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:29.5158767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81205 2022-03-17T19:54:29.5282673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81206 2022-03-17T19:54:30.3402257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:30.3410196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:32.0028263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:32.0028805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:32.3350989Z ok (4.425s) 2022-03-17T19:54:32.3351175Z 2022-03-17T19:54:32.3351607Z ---------------------------------------------------------------------- 2022-03-17T19:54:32.3351934Z Ran 1 test in 4.426s 2022-03-17T19:54:32.3352100Z 2022-03-17T19:54:32.3352179Z OK 2022-03-17T19:54:32.3352284Z 2022-03-17T19:54:32.3352421Z Generating XML reports... 2022-03-17T19:54:32.3395331Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195427.xml 2022-03-17T19:54:33.4005249Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:33.4018710Z 2022-03-17T19:54:33.4019092Z Running tests... 2022-03-17T19:54:33.4019627Z ---------------------------------------------------------------------- 2022-03-17T19:54:34.9395387Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:34.9825517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81326 2022-03-17T19:54:34.9949746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81327 2022-03-17T19:54:35.8027427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:35.8104943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:35.9985673Z skip (2.596s) 2022-03-17T19:54:35.9985867Z 2022-03-17T19:54:35.9986581Z ---------------------------------------------------------------------- 2022-03-17T19:54:35.9986978Z Ran 1 test in 2.597s 2022-03-17T19:54:35.9987145Z 2022-03-17T19:54:35.9987261Z OK (skipped=1) 2022-03-17T19:54:35.9987397Z 2022-03-17T19:54:35.9987523Z Generating XML reports... 2022-03-17T19:54:36.0031459Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195433.xml 2022-03-17T19:54:37.0430092Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:37.0444085Z 2022-03-17T19:54:37.0444366Z Running tests... 2022-03-17T19:54:37.0444817Z ---------------------------------------------------------------------- 2022-03-17T19:54:38.5971149Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:38.6394561Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81435 2022-03-17T19:54:38.6519487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81436 2022-03-17T19:54:39.4814301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:39.4952813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:39.6555229Z skip (2.611s) 2022-03-17T19:54:39.6555471Z 2022-03-17T19:54:39.6555844Z ---------------------------------------------------------------------- 2022-03-17T19:54:39.6556184Z Ran 1 test in 2.611s 2022-03-17T19:54:39.6556348Z 2022-03-17T19:54:39.6556474Z OK (skipped=1) 2022-03-17T19:54:39.6556611Z 2022-03-17T19:54:39.6556737Z Generating XML reports... 2022-03-17T19:54:39.6598937Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195437.xml 2022-03-17T19:54:40.7077076Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:40.7090669Z 2022-03-17T19:54:40.7090793Z Running tests... 2022-03-17T19:54:40.7091456Z ---------------------------------------------------------------------- 2022-03-17T19:54:42.2556789Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:42.2987709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81544 2022-03-17T19:54:42.3109987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81545 2022-03-17T19:54:43.1120321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:43.1312188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:43.1521461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:43.1521942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:43.3145614Z ok (2.605s) 2022-03-17T19:54:43.3145803Z 2022-03-17T19:54:43.3146172Z ---------------------------------------------------------------------- 2022-03-17T19:54:43.3146525Z Ran 1 test in 2.605s 2022-03-17T19:54:43.3146681Z 2022-03-17T19:54:43.3146766Z OK 2022-03-17T19:54:43.3146858Z 2022-03-17T19:54:43.3146987Z Generating XML reports... 2022-03-17T19:54:43.3189395Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195440.xml 2022-03-17T19:54:44.3693409Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:44.3708072Z 2022-03-17T19:54:44.3708347Z Running tests... 2022-03-17T19:54:44.3708770Z ---------------------------------------------------------------------- 2022-03-17T19:54:45.9158458Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:45.9586776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81663 2022-03-17T19:54:45.9709330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81664 2022-03-17T19:54:46.8081592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:46.8204510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:46.8485364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:46.8486487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:47.0749406Z ok (2.704s) 2022-03-17T19:54:47.0749783Z 2022-03-17T19:54:47.0750508Z ---------------------------------------------------------------------- 2022-03-17T19:54:47.0751151Z Ran 1 test in 2.704s 2022-03-17T19:54:47.0751354Z 2022-03-17T19:54:47.0751447Z OK 2022-03-17T19:54:47.0751591Z 2022-03-17T19:54:47.0751723Z Generating XML reports... 2022-03-17T19:54:47.0794153Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195444.xml 2022-03-17T19:54:48.0826097Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:48.0840462Z 2022-03-17T19:54:48.0841001Z Running tests... 2022-03-17T19:54:48.0841805Z ---------------------------------------------------------------------- 2022-03-17T19:54:48.0861569Z test_ignored_output (__main__.DistributedDataParallelTest) 2022-03-17T19:54:49.6223286Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:49.6649570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81782 2022-03-17T19:54:49.6769899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81783 2022-03-17T19:54:50.5382422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:50.5464655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:50.5698161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:50.5698619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:54:50.7808193Z ok (2.696s) 2022-03-17T19:54:50.7808451Z 2022-03-17T19:54:50.7808813Z ---------------------------------------------------------------------- 2022-03-17T19:54:50.7809144Z Ran 1 test in 2.697s 2022-03-17T19:54:50.7809306Z 2022-03-17T19:54:50.7809398Z OK 2022-03-17T19:54:50.7809533Z 2022-03-17T19:54:50.7809654Z Generating XML reports... 2022-03-17T19:54:50.7852379Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195448.xml 2022-03-17T19:54:51.8309944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:51.8323159Z 2022-03-17T19:54:51.8323558Z Running tests... 2022-03-17T19:54:51.8324055Z ---------------------------------------------------------------------- 2022-03-17T19:54:51.8344961Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest) 2022-03-17T19:54:53.3769959Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:53.4198113Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81931 2022-03-17T19:54:53.4320020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81932 2022-03-17T19:54:54.2373467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:54.2706069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:54.5357882Z ok (2.703s) 2022-03-17T19:54:54.5358126Z 2022-03-17T19:54:54.5358517Z ---------------------------------------------------------------------- 2022-03-17T19:54:54.5358857Z Ran 1 test in 2.703s 2022-03-17T19:54:54.5358997Z 2022-03-17T19:54:54.5359080Z OK 2022-03-17T19:54:54.5359214Z 2022-03-17T19:54:54.5359342Z Generating XML reports... 2022-03-17T19:54:54.5402859Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195451.xml 2022-03-17T19:54:55.5913063Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:55.5927530Z 2022-03-17T19:54:55.5927734Z Running tests... 2022-03-17T19:54:55.5928391Z ---------------------------------------------------------------------- 2022-03-17T19:54:57.1283121Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:54:57.1710945Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82080 2022-03-17T19:54:57.1832560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82081 2022-03-17T19:54:57.9829960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:54:57.9835761Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:57.9837318Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:57.9838299Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:57.9839265Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:57.9840255Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:57.9841223Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:58.0243193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:54:58.0249369Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:58.0250781Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:58.0251771Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:58.0253136Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:58.0254097Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:58.0255314Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T19:54:58.1868308Z ok (2.594s) 2022-03-17T19:54:58.1868576Z 2022-03-17T19:54:58.1868972Z ---------------------------------------------------------------------- 2022-03-17T19:54:58.1869311Z Ran 1 test in 2.594s 2022-03-17T19:54:58.1869456Z 2022-03-17T19:54:58.1869548Z OK 2022-03-17T19:54:58.1869682Z 2022-03-17T19:54:58.1870084Z Generating XML reports... 2022-03-17T19:54:58.1913062Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195455.xml 2022-03-17T19:54:59.2406210Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:54:59.2420210Z 2022-03-17T19:54:59.2420599Z Running tests... 2022-03-17T19:54:59.2421414Z ---------------------------------------------------------------------- 2022-03-17T19:55:00.7849161Z test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:00.8280534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82189 2022-03-17T19:55:00.8405704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82190 2022-03-17T19:55:01.6499315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:01.7015736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:01.7125448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:55:01.7126136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:55:01.7127178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:01.7127897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:55:03.3526919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:55:03.3527514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:55:03.3565492Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T19:55:03.3566309Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T19:55:03.3676321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:55:03.3676848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T19:55:03.6474841Z ok (4.405s) 2022-03-17T19:55:03.6475067Z 2022-03-17T19:55:03.6475470Z ---------------------------------------------------------------------- 2022-03-17T19:55:03.6475778Z Ran 1 test in 4.405s 2022-03-17T19:55:03.6475942Z 2022-03-17T19:55:03.6476013Z OK 2022-03-17T19:55:03.6476145Z 2022-03-17T19:55:03.6476276Z Generating XML reports... 2022-03-17T19:55:03.6518820Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195459.xml 2022-03-17T19:55:04.7002564Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:04.7015810Z 2022-03-17T19:55:04.7016229Z Running tests... 2022-03-17T19:55:04.7017068Z ---------------------------------------------------------------------- 2022-03-17T19:55:06.2224709Z test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:06.2646091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82310 2022-03-17T19:55:06.2767846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82311 2022-03-17T19:55:07.1094386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:07.1352570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:07.3804911Z ok (2.679s) 2022-03-17T19:55:07.3805131Z 2022-03-17T19:55:07.3805800Z ---------------------------------------------------------------------- 2022-03-17T19:55:07.3806153Z Ran 1 test in 2.679s 2022-03-17T19:55:07.3806322Z 2022-03-17T19:55:07.3806422Z OK 2022-03-17T19:55:07.3806557Z 2022-03-17T19:55:07.3806672Z Generating XML reports... 2022-03-17T19:55:07.3849514Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195504.xml 2022-03-17T19:55:08.4252844Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:08.4266389Z 2022-03-17T19:55:08.4266650Z Running tests... 2022-03-17T19:55:08.4267353Z ---------------------------------------------------------------------- 2022-03-17T19:55:09.9545641Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:09.9972024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82459 2022-03-17T19:55:10.0095038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82460 2022-03-17T19:55:10.8109183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:10.8345598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:11.0129970Z ok (2.586s) 2022-03-17T19:55:11.0130200Z 2022-03-17T19:55:11.0130576Z ---------------------------------------------------------------------- 2022-03-17T19:55:11.0130916Z Ran 1 test in 2.586s 2022-03-17T19:55:11.0131070Z 2022-03-17T19:55:11.0131174Z OK 2022-03-17T19:55:11.0131293Z 2022-03-17T19:55:11.0131425Z Generating XML reports... 2022-03-17T19:55:11.0173906Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195508.xml 2022-03-17T19:55:12.0609457Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:12.0625508Z 2022-03-17T19:55:12.0626212Z Running tests... 2022-03-17T19:55:12.0626710Z ---------------------------------------------------------------------- 2022-03-17T19:55:13.5920914Z test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:13.6353429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82608 2022-03-17T19:55:13.6479753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82609 2022-03-17T19:55:13.6605529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82610 2022-03-17T19:55:13.6736038Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82611 2022-03-17T19:55:14.4695789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:14.5232747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:14.5337690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:14.5400106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:14.7776859Z ok (2.715s) 2022-03-17T19:55:14.7777087Z 2022-03-17T19:55:14.7777468Z ---------------------------------------------------------------------- 2022-03-17T19:55:14.7778122Z Ran 1 test in 2.715s 2022-03-17T19:55:14.7778289Z 2022-03-17T19:55:14.7778389Z OK 2022-03-17T19:55:14.7778507Z 2022-03-17T19:55:14.7778644Z Generating XML reports... 2022-03-17T19:55:14.7822614Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195512.xml 2022-03-17T19:55:15.8294871Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:15.8309811Z 2022-03-17T19:55:15.8309935Z Running tests... 2022-03-17T19:55:15.8310856Z ---------------------------------------------------------------------- 2022-03-17T19:55:17.3867318Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:17.4296119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82801 2022-03-17T19:55:17.4422628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82802 2022-03-17T19:55:17.4546346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82803 2022-03-17T19:55:17.4674401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82804 2022-03-17T19:55:18.2790179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:18.3069949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:18.3073271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:18.3140051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:20.2758856Z ok (4.445s) 2022-03-17T19:55:20.2759163Z 2022-03-17T19:55:20.2759554Z ---------------------------------------------------------------------- 2022-03-17T19:55:20.2759891Z Ran 1 test in 4.445s 2022-03-17T19:55:20.2760038Z 2022-03-17T19:55:20.2760136Z OK 2022-03-17T19:55:20.2760273Z 2022-03-17T19:55:20.2760418Z Generating XML reports... 2022-03-17T19:55:20.2802640Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195515.xml 2022-03-17T19:55:21.3342561Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:21.3356829Z 2022-03-17T19:55:21.3356989Z Running tests... 2022-03-17T19:55:21.3357842Z ---------------------------------------------------------------------- 2022-03-17T19:55:22.8724536Z test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:22.9153080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82998 2022-03-17T19:55:22.9278133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82999 2022-03-17T19:55:22.9406253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83000 2022-03-17T19:55:22.9537979Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83001 2022-03-17T19:55:23.7862099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:23.7869150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:23.8056957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:23.8080921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:24.0578743Z ok (2.722s) 2022-03-17T19:55:24.0579139Z 2022-03-17T19:55:24.0579535Z ---------------------------------------------------------------------- 2022-03-17T19:55:24.0579820Z Ran 1 test in 2.722s 2022-03-17T19:55:24.0579986Z 2022-03-17T19:55:24.0580096Z OK 2022-03-17T19:55:24.0581246Z 2022-03-17T19:55:24.0581404Z Generating XML reports... 2022-03-17T19:55:24.0624108Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195521.xml 2022-03-17T19:55:25.0980044Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:25.0994540Z 2022-03-17T19:55:25.0994783Z Running tests... 2022-03-17T19:55:25.0995212Z ---------------------------------------------------------------------- 2022-03-17T19:55:26.6527514Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:26.6956482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83191 2022-03-17T19:55:26.7081788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83192 2022-03-17T19:55:26.7205802Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83193 2022-03-17T19:55:26.7334107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83194 2022-03-17T19:55:27.5267589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:27.5276283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:27.5303877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:27.5961988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:27.6119682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:55:27.6221405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:55:27.6323540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:55:27.6324715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:55:27.6325676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:55:27.6326351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:55:27.6327040Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:55:27.6327721Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:55:27.8373914Z ok (2.738s) 2022-03-17T19:55:27.8374281Z 2022-03-17T19:55:27.8375011Z ---------------------------------------------------------------------- 2022-03-17T19:55:27.8375649Z Ran 1 test in 2.738s 2022-03-17T19:55:27.8375820Z 2022-03-17T19:55:27.8375964Z OK 2022-03-17T19:55:27.8376112Z 2022-03-17T19:55:27.8378054Z Generating XML reports... 2022-03-17T19:55:27.8419166Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195525.xml 2022-03-17T19:55:28.8842852Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:28.8856974Z 2022-03-17T19:55:28.8857294Z Running tests... 2022-03-17T19:55:28.8857982Z ---------------------------------------------------------------------- 2022-03-17T19:55:30.4397250Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:30.4833085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83384 2022-03-17T19:55:30.4962738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83385 2022-03-17T19:55:30.5092649Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83386 2022-03-17T19:55:30.5220197Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83387 2022-03-17T19:55:31.3461950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:31.3497672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:31.3845982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:31.4006771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:31.6262639Z ok (2.740s) 2022-03-17T19:55:31.6263118Z 2022-03-17T19:55:31.6263523Z ---------------------------------------------------------------------- 2022-03-17T19:55:31.6263863Z Ran 1 test in 2.741s 2022-03-17T19:55:31.6264029Z 2022-03-17T19:55:31.6264121Z OK 2022-03-17T19:55:31.6264260Z 2022-03-17T19:55:31.6264393Z Generating XML reports... 2022-03-17T19:55:31.6307433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195528.xml 2022-03-17T19:55:32.6693984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:32.6708951Z 2022-03-17T19:55:32.6709107Z Running tests... 2022-03-17T19:55:32.6709603Z ---------------------------------------------------------------------- 2022-03-17T19:55:34.2072352Z test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:34.2500093Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83577 2022-03-17T19:55:34.2622622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83578 2022-03-17T19:55:34.2747777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83579 2022-03-17T19:55:34.2881882Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83580 2022-03-17T19:55:35.1444845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:35.1644787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:35.1650867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:35.1860705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:35.9934011Z ok (3.322s) 2022-03-17T19:55:35.9934304Z 2022-03-17T19:55:35.9934717Z ---------------------------------------------------------------------- 2022-03-17T19:55:35.9935060Z Ran 1 test in 3.323s 2022-03-17T19:55:35.9935226Z 2022-03-17T19:55:35.9935286Z OK 2022-03-17T19:55:35.9935432Z 2022-03-17T19:55:35.9935517Z Generating XML reports... 2022-03-17T19:55:35.9978710Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195532.xml 2022-03-17T19:55:37.0588219Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:37.0601754Z 2022-03-17T19:55:37.0602184Z Running tests... 2022-03-17T19:55:37.0602683Z ---------------------------------------------------------------------- 2022-03-17T19:55:38.5961057Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:38.6389884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83794 2022-03-17T19:55:38.6512756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83795 2022-03-17T19:55:38.6637474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83796 2022-03-17T19:55:38.6770205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83797 2022-03-17T19:55:39.4680707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:39.4891516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:39.5054436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:39.5335734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:43.3876766Z ok (6.327s) 2022-03-17T19:55:43.3876972Z 2022-03-17T19:55:43.3877902Z ---------------------------------------------------------------------- 2022-03-17T19:55:43.3878629Z Ran 1 test in 6.327s 2022-03-17T19:55:43.3878800Z 2022-03-17T19:55:43.3878888Z OK 2022-03-17T19:55:43.3879027Z 2022-03-17T19:55:43.3879181Z Generating XML reports... 2022-03-17T19:55:43.3931944Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195537.xml 2022-03-17T19:55:44.4554202Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:44.4568417Z 2022-03-17T19:55:44.4568800Z Running tests... 2022-03-17T19:55:44.4569428Z ---------------------------------------------------------------------- 2022-03-17T19:55:45.9887154Z test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:46.0316989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84015 2022-03-17T19:55:46.0441158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84016 2022-03-17T19:55:46.0566111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84017 2022-03-17T19:55:46.0694181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84018 2022-03-17T19:55:46.8866467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:46.9281457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:46.9302431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:46.9517497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:47.2736821Z ok (2.816s) 2022-03-17T19:55:47.2737058Z 2022-03-17T19:55:47.2737550Z ---------------------------------------------------------------------- 2022-03-17T19:55:47.2737939Z Ran 1 test in 2.817s 2022-03-17T19:55:47.2738084Z 2022-03-17T19:55:47.2738187Z OK 2022-03-17T19:55:47.2738322Z 2022-03-17T19:55:47.2738429Z Generating XML reports... 2022-03-17T19:55:47.2781740Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195544.xml 2022-03-17T19:55:48.3203468Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:48.3218188Z 2022-03-17T19:55:48.3218387Z Running tests... 2022-03-17T19:55:48.3218825Z ---------------------------------------------------------------------- 2022-03-17T19:55:49.8554694Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:49.8990039Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84208 2022-03-17T19:55:49.9113389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84209 2022-03-17T19:55:49.9239287Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84210 2022-03-17T19:55:49.9372587Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84211 2022-03-17T19:55:50.7555603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:50.7833572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:50.8307281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:50.8580263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:52.7444780Z ok (4.422s) 2022-03-17T19:55:52.7445051Z 2022-03-17T19:55:52.7445469Z ---------------------------------------------------------------------- 2022-03-17T19:55:52.7445810Z Ran 1 test in 4.423s 2022-03-17T19:55:52.7445933Z 2022-03-17T19:55:52.7446023Z OK 2022-03-17T19:55:52.7446168Z 2022-03-17T19:55:52.7446602Z Generating XML reports... 2022-03-17T19:55:52.7489374Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195548.xml 2022-03-17T19:55:53.8031905Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:53.8045628Z 2022-03-17T19:55:53.8046054Z Running tests... 2022-03-17T19:55:53.8046536Z ---------------------------------------------------------------------- 2022-03-17T19:55:55.3290880Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:55:55.3719313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84405 2022-03-17T19:55:55.3840395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84406 2022-03-17T19:55:55.3966827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84407 2022-03-17T19:55:55.4093881Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84408 2022-03-17T19:55:56.3184081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:55:56.3193395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:55:56.3228711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:55:56.3912021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:55:58.3170920Z ok (4.512s) 2022-03-17T19:55:58.3171266Z 2022-03-17T19:55:58.3171634Z ---------------------------------------------------------------------- 2022-03-17T19:55:58.3171980Z Ran 1 test in 4.512s 2022-03-17T19:55:58.3172078Z 2022-03-17T19:55:58.3172157Z OK 2022-03-17T19:55:58.3172295Z 2022-03-17T19:55:58.3172443Z Generating XML reports... 2022-03-17T19:55:58.3215050Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195553.xml 2022-03-17T19:55:59.3810950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:55:59.3825545Z 2022-03-17T19:55:59.3825839Z Running tests... 2022-03-17T19:55:59.3826296Z ---------------------------------------------------------------------- 2022-03-17T19:56:00.9178853Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:00.9609871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84602 2022-03-17T19:56:00.9731987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84603 2022-03-17T19:56:00.9857091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84604 2022-03-17T19:56:00.9990613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84605 2022-03-17T19:56:01.7992216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:01.8445199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:01.8645489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:01.8721070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:02.1031012Z ok (2.720s) 2022-03-17T19:56:02.1031221Z 2022-03-17T19:56:02.1031614Z ---------------------------------------------------------------------- 2022-03-17T19:56:02.1031962Z Ran 1 test in 2.721s 2022-03-17T19:56:02.1032126Z 2022-03-17T19:56:02.1032219Z OK 2022-03-17T19:56:02.1032353Z 2022-03-17T19:56:02.1032460Z Generating XML reports... 2022-03-17T19:56:02.1076230Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195559.xml 2022-03-17T19:56:03.1707995Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:03.1722249Z 2022-03-17T19:56:03.1722411Z Running tests... 2022-03-17T19:56:03.1723169Z ---------------------------------------------------------------------- 2022-03-17T19:56:04.7190490Z test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:04.7622688Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84795 2022-03-17T19:56:04.7746940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84796 2022-03-17T19:56:04.7876018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84797 2022-03-17T19:56:04.8001409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84798 2022-03-17T19:56:05.6262270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:05.6449595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:05.6730572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:05.7028073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:05.9042895Z ok (2.732s) 2022-03-17T19:56:05.9043286Z 2022-03-17T19:56:05.9044051Z ---------------------------------------------------------------------- 2022-03-17T19:56:05.9044506Z Ran 1 test in 2.732s 2022-03-17T19:56:05.9044679Z 2022-03-17T19:56:05.9044770Z OK 2022-03-17T19:56:05.9044906Z 2022-03-17T19:56:05.9045039Z Generating XML reports... 2022-03-17T19:56:05.9087061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195603.xml 2022-03-17T19:56:06.9465247Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:06.9478779Z 2022-03-17T19:56:06.9479276Z Running tests... 2022-03-17T19:56:06.9480226Z ---------------------------------------------------------------------- 2022-03-17T19:56:08.4738896Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:08.5168754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84988 2022-03-17T19:56:08.5291709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84989 2022-03-17T19:56:08.5416981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84990 2022-03-17T19:56:08.5545138Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84991 2022-03-17T19:56:09.4229229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:09.4268488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:09.4399862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:09.4450109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:09.4611326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:56:09.4714186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:56:09.4817846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3 2022-03-17T19:56:09.4818358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2 2022-03-17T19:56:09.4819118Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:56:09.4819807Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:56:09.4918998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:56:09.4919700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. 2022-03-17T19:56:09.7615005Z ok (2.813s) 2022-03-17T19:56:09.7615182Z 2022-03-17T19:56:09.7615577Z ---------------------------------------------------------------------- 2022-03-17T19:56:09.7616246Z Ran 1 test in 2.814s 2022-03-17T19:56:09.7616414Z 2022-03-17T19:56:09.7616506Z OK 2022-03-17T19:56:09.7616635Z 2022-03-17T19:56:09.7616764Z Generating XML reports... 2022-03-17T19:56:09.7658313Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195606.xml 2022-03-17T19:56:10.8081783Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:10.8095312Z 2022-03-17T19:56:10.8095582Z Running tests... 2022-03-17T19:56:10.8095973Z ---------------------------------------------------------------------- 2022-03-17T19:56:12.3518475Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:12.3947712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85181 2022-03-17T19:56:12.4070304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85182 2022-03-17T19:56:12.4198805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85183 2022-03-17T19:56:12.4326829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85184 2022-03-17T19:56:13.3420821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:13.3573945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:13.3833237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:13.4275826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:13.6368628Z ok (2.827s) 2022-03-17T19:56:13.6368865Z 2022-03-17T19:56:13.6369296Z ---------------------------------------------------------------------- 2022-03-17T19:56:13.6369632Z Ran 1 test in 2.827s 2022-03-17T19:56:13.6369797Z 2022-03-17T19:56:13.6369887Z OK 2022-03-17T19:56:13.6369967Z 2022-03-17T19:56:13.6370114Z Generating XML reports... 2022-03-17T19:56:13.6413581Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195610.xml 2022-03-17T19:56:14.6846436Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:14.6860588Z 2022-03-17T19:56:14.6860993Z Running tests... 2022-03-17T19:56:14.6861500Z ---------------------------------------------------------------------- 2022-03-17T19:56:16.2232222Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:16.2660092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85374 2022-03-17T19:56:16.2781191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85375 2022-03-17T19:56:16.2908067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85376 2022-03-17T19:56:16.3039998Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85377 2022-03-17T19:56:17.1615064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:17.1898958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:17.1948276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:17.1972824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:17.5082479Z ok (2.822s) 2022-03-17T19:56:17.5082643Z 2022-03-17T19:56:17.5083591Z ---------------------------------------------------------------------- 2022-03-17T19:56:17.5083994Z Ran 1 test in 2.822s 2022-03-17T19:56:17.5084158Z 2022-03-17T19:56:17.5084222Z OK 2022-03-17T19:56:17.5084400Z 2022-03-17T19:56:17.5084529Z Generating XML reports... 2022-03-17T19:56:17.5129049Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195614.xml 2022-03-17T19:56:18.5468707Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:18.5482739Z 2022-03-17T19:56:18.5482885Z Running tests... 2022-03-17T19:56:18.5483558Z ---------------------------------------------------------------------- 2022-03-17T19:56:20.0794838Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:20.1220053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85567 2022-03-17T19:56:20.1342051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85568 2022-03-17T19:56:20.1466979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85569 2022-03-17T19:56:20.1597326Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85570 2022-03-17T19:56:20.9974996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:21.0152239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:21.0161990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:21.0172045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:22.9671497Z ok (4.418s) 2022-03-17T19:56:22.9671741Z 2022-03-17T19:56:22.9672160Z ---------------------------------------------------------------------- 2022-03-17T19:56:22.9672480Z Ran 1 test in 4.419s 2022-03-17T19:56:22.9672630Z 2022-03-17T19:56:22.9672729Z OK 2022-03-17T19:56:22.9672884Z 2022-03-17T19:56:22.9672991Z Generating XML reports... 2022-03-17T19:56:22.9719817Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195618.xml 2022-03-17T19:56:24.0295565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:24.0311486Z 2022-03-17T19:56:24.0311897Z Running tests... 2022-03-17T19:56:24.0312395Z ---------------------------------------------------------------------- 2022-03-17T19:56:25.5614818Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:25.6045482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85764 2022-03-17T19:56:25.6168482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85765 2022-03-17T19:56:25.6295847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85766 2022-03-17T19:56:25.6424832Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85767 2022-03-17T19:56:26.4605291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:26.4622390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:26.4880946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:26.4910126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:27.1472617Z ok (3.116s) 2022-03-17T19:56:27.1472884Z 2022-03-17T19:56:27.1473274Z ---------------------------------------------------------------------- 2022-03-17T19:56:27.1473611Z Ran 1 test in 3.116s 2022-03-17T19:56:27.1473760Z 2022-03-17T19:56:27.1473864Z OK 2022-03-17T19:56:27.1473999Z 2022-03-17T19:56:27.1474136Z Generating XML reports... 2022-03-17T19:56:27.1517467Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195624.xml 2022-03-17T19:56:28.2049115Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:28.2063605Z 2022-03-17T19:56:28.2064023Z Running tests... 2022-03-17T19:56:28.2064532Z ---------------------------------------------------------------------- 2022-03-17T19:56:29.7389873Z test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:29.7820349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85981 2022-03-17T19:56:29.7940543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85982 2022-03-17T19:56:29.8065393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85983 2022-03-17T19:56:29.8197138Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85984 2022-03-17T19:56:30.6574145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:30.6585602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:30.6688522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:30.6854466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:31.1240689Z ok (2.917s) 2022-03-17T19:56:31.1240971Z 2022-03-17T19:56:31.1241352Z ---------------------------------------------------------------------- 2022-03-17T19:56:31.1241693Z Ran 1 test in 2.918s 2022-03-17T19:56:31.1241835Z 2022-03-17T19:56:31.1241858Z OK 2022-03-17T19:56:31.1241982Z 2022-03-17T19:56:31.1242118Z Generating XML reports... 2022-03-17T19:56:31.1285748Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195628.xml 2022-03-17T19:56:32.1769201Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:32.1783392Z 2022-03-17T19:56:32.1783884Z Running tests... 2022-03-17T19:56:32.1784349Z ---------------------------------------------------------------------- 2022-03-17T19:56:33.7199274Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:33.7627931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86198 2022-03-17T19:56:33.7750297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86199 2022-03-17T19:56:33.7879549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86200 2022-03-17T19:56:33.8007517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86201 2022-03-17T19:56:34.6489372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:34.6619573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:34.6724952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:34.6774926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:37.1088883Z ok (4.930s) 2022-03-17T19:56:37.1089116Z 2022-03-17T19:56:37.1089552Z ---------------------------------------------------------------------- 2022-03-17T19:56:37.1089794Z Ran 1 test in 4.930s 2022-03-17T19:56:37.1089961Z 2022-03-17T19:56:37.1090073Z OK 2022-03-17T19:56:37.1090210Z 2022-03-17T19:56:37.1090331Z Generating XML reports... 2022-03-17T19:56:37.1133418Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195632.xml 2022-03-17T19:56:38.1713773Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:38.1728230Z 2022-03-17T19:56:38.1728703Z Running tests... 2022-03-17T19:56:38.1729144Z ---------------------------------------------------------------------- 2022-03-17T19:56:39.7072171Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:39.7501511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86419 2022-03-17T19:56:39.7621802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86420 2022-03-17T19:56:39.7746304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86421 2022-03-17T19:56:39.7877536Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86422 2022-03-17T19:56:40.5844534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:40.5880864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:40.6422503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:40.6545324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:40.8918190Z ok (2.719s) 2022-03-17T19:56:40.8918344Z 2022-03-17T19:56:40.8919092Z ---------------------------------------------------------------------- 2022-03-17T19:56:40.8919374Z Ran 1 test in 2.719s 2022-03-17T19:56:40.8919544Z 2022-03-17T19:56:40.8919646Z OK 2022-03-17T19:56:40.8919841Z 2022-03-17T19:56:40.8919902Z Generating XML reports... 2022-03-17T19:56:40.8962356Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195638.xml 2022-03-17T19:56:41.9407592Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:41.9421575Z 2022-03-17T19:56:41.9422084Z Running tests... 2022-03-17T19:56:41.9422593Z ---------------------------------------------------------------------- 2022-03-17T19:56:43.4758650Z test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:43.5199275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86612 2022-03-17T19:56:43.5324160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86613 2022-03-17T19:56:43.5452553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86614 2022-03-17T19:56:43.5587809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86615 2022-03-17T19:56:44.3378062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:44.3511594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:44.3804712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:44.4070507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:44.6628193Z ok (2.720s) 2022-03-17T19:56:44.6628475Z 2022-03-17T19:56:44.6628889Z ---------------------------------------------------------------------- 2022-03-17T19:56:44.6629218Z Ran 1 test in 2.721s 2022-03-17T19:56:44.6629289Z 2022-03-17T19:56:44.6629438Z OK 2022-03-17T19:56:44.6629547Z 2022-03-17T19:56:44.6629677Z Generating XML reports... 2022-03-17T19:56:44.6672534Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195641.xml 2022-03-17T19:56:45.7169687Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:45.7186387Z 2022-03-17T19:56:45.7186674Z Running tests... 2022-03-17T19:56:45.7187104Z ---------------------------------------------------------------------- 2022-03-17T19:56:47.2698037Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:47.3125557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86805 2022-03-17T19:56:47.3250989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86806 2022-03-17T19:56:47.3377980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86807 2022-03-17T19:56:47.3506336Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86808 2022-03-17T19:56:48.1852752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:48.2299092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:48.2306732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:48.2311996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:50.1580528Z ok (4.439s) 2022-03-17T19:56:50.1581039Z 2022-03-17T19:56:50.1581437Z ---------------------------------------------------------------------- 2022-03-17T19:56:50.1581793Z Ran 1 test in 4.439s 2022-03-17T19:56:50.1581971Z 2022-03-17T19:56:50.1582055Z OK 2022-03-17T19:56:50.1582166Z 2022-03-17T19:56:50.1582300Z Generating XML reports... 2022-03-17T19:56:50.1629222Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195645.xml 2022-03-17T19:56:51.2328484Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:51.2342438Z 2022-03-17T19:56:51.2342632Z Running tests... 2022-03-17T19:56:51.2343100Z ---------------------------------------------------------------------- 2022-03-17T19:56:52.7732607Z test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:52.8163934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87002 2022-03-17T19:56:52.8287307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87003 2022-03-17T19:56:52.8415606Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87004 2022-03-17T19:56:52.8551875Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87005 2022-03-17T19:56:53.6674323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:53.6851936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:53.7088441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:53.7227216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:54.0593235Z ok (2.825s) 2022-03-17T19:56:54.0593451Z 2022-03-17T19:56:54.0593870Z ---------------------------------------------------------------------- 2022-03-17T19:56:54.0594192Z Ran 1 test in 2.825s 2022-03-17T19:56:54.0594362Z 2022-03-17T19:56:54.0595787Z OK 2022-03-17T19:56:54.0595908Z 2022-03-17T19:56:54.0596389Z Generating XML reports... 2022-03-17T19:56:54.0638705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195651.xml 2022-03-17T19:56:55.1073854Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:55.1088152Z 2022-03-17T19:56:55.1088407Z Running tests... 2022-03-17T19:56:55.1088841Z ---------------------------------------------------------------------- 2022-03-17T19:56:56.6410950Z test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:56:56.6842222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87195 2022-03-17T19:56:56.6964996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87196 2022-03-17T19:56:56.7092032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87197 2022-03-17T19:56:56.7227768Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87198 2022-03-17T19:56:57.5363447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:56:57.5366009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:56:57.5876046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:56:57.5946550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:56:57.9269697Z ok (2.818s) 2022-03-17T19:56:57.9270078Z 2022-03-17T19:56:57.9270827Z ---------------------------------------------------------------------- 2022-03-17T19:56:57.9271525Z Ran 1 test in 2.818s 2022-03-17T19:56:57.9271672Z 2022-03-17T19:56:57.9271764Z OK 2022-03-17T19:56:57.9271899Z 2022-03-17T19:56:57.9272039Z Generating XML reports... 2022-03-17T19:56:57.9314070Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195655.xml 2022-03-17T19:56:58.9754198Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:56:58.9768438Z 2022-03-17T19:56:58.9768721Z Running tests... 2022-03-17T19:56:58.9769623Z ---------------------------------------------------------------------- 2022-03-17T19:57:00.5129216Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:00.5558379Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87412 2022-03-17T19:57:00.5681505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87413 2022-03-17T19:57:00.5807823Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87414 2022-03-17T19:57:00.5939517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87415 2022-03-17T19:57:01.4880358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:01.4928101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:01.4929623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:01.4934036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:03.9017613Z ok (4.925s) 2022-03-17T19:57:03.9017820Z 2022-03-17T19:57:03.9018280Z ---------------------------------------------------------------------- 2022-03-17T19:57:03.9018602Z Ran 1 test in 4.925s 2022-03-17T19:57:03.9018750Z 2022-03-17T19:57:03.9018844Z OK 2022-03-17T19:57:03.9019007Z 2022-03-17T19:57:03.9019136Z Generating XML reports... 2022-03-17T19:57:03.9061185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195658.xml 2022-03-17T19:57:04.9639699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:04.9654027Z 2022-03-17T19:57:04.9654196Z Running tests... 2022-03-17T19:57:04.9654615Z ---------------------------------------------------------------------- 2022-03-17T19:57:06.5100785Z test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:06.5530452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87633 2022-03-17T19:57:06.5656563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87634 2022-03-17T19:57:06.5786859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87635 2022-03-17T19:57:06.5913062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87636 2022-03-17T19:57:07.3770241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:07.3861711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:07.3897514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:07.4452477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:07.6955021Z ok (2.730s) 2022-03-17T19:57:07.6955218Z 2022-03-17T19:57:07.6956136Z ---------------------------------------------------------------------- 2022-03-17T19:57:07.6956763Z Ran 1 test in 2.730s 2022-03-17T19:57:07.6957035Z 2022-03-17T19:57:07.6957196Z OK 2022-03-17T19:57:07.6957480Z 2022-03-17T19:57:07.6957718Z Generating XML reports... 2022-03-17T19:57:07.7001728Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195704.xml 2022-03-17T19:57:08.7119136Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:08.7133820Z 2022-03-17T19:57:08.7134259Z Running tests... 2022-03-17T19:57:08.7134714Z ---------------------------------------------------------------------- 2022-03-17T19:57:10.2582660Z test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:10.3017242Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87826 2022-03-17T19:57:10.3139755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87827 2022-03-17T19:57:10.3270649Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87828 2022-03-17T19:57:10.3401676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87829 2022-03-17T19:57:11.1605391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:11.2018851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:11.2216461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:11.2262036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:11.4442773Z ok (2.731s) 2022-03-17T19:57:11.4443023Z 2022-03-17T19:57:11.4443364Z ---------------------------------------------------------------------- 2022-03-17T19:57:11.4443710Z Ran 1 test in 2.731s 2022-03-17T19:57:11.4443876Z 2022-03-17T19:57:11.4443949Z OK 2022-03-17T19:57:11.4444083Z 2022-03-17T19:57:11.4444220Z Generating XML reports... 2022-03-17T19:57:11.4487789Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195708.xml 2022-03-17T19:57:12.4892742Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:12.4907268Z 2022-03-17T19:57:12.4907545Z Running tests... 2022-03-17T19:57:12.4907989Z ---------------------------------------------------------------------- 2022-03-17T19:57:14.0182173Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:14.0607297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88019 2022-03-17T19:57:14.0731013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88020 2022-03-17T19:57:14.0856393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88021 2022-03-17T19:57:14.0987227Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88022 2022-03-17T19:57:14.9399578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:14.9426902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:14.9591001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:14.9878603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:16.9059072Z ok (4.415s) 2022-03-17T19:57:16.9059256Z 2022-03-17T19:57:16.9059693Z ---------------------------------------------------------------------- 2022-03-17T19:57:16.9060015Z Ran 1 test in 4.415s 2022-03-17T19:57:16.9060170Z 2022-03-17T19:57:16.9060502Z OK 2022-03-17T19:57:16.9060631Z 2022-03-17T19:57:16.9060783Z Generating XML reports... 2022-03-17T19:57:16.9103628Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195712.xml 2022-03-17T19:57:17.9730184Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:17.9745239Z 2022-03-17T19:57:17.9745566Z Running tests... 2022-03-17T19:57:17.9745999Z ---------------------------------------------------------------------- 2022-03-17T19:57:19.5312851Z test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:19.5754345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88216 2022-03-17T19:57:19.5886361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88217 2022-03-17T19:57:19.6021517Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88218 2022-03-17T19:57:19.6159354Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88219 2022-03-17T19:57:20.4356441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:20.4535755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:20.4965076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:20.4984000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:20.8203252Z ok (2.845s) 2022-03-17T19:57:20.8203627Z 2022-03-17T19:57:20.8204354Z ---------------------------------------------------------------------- 2022-03-17T19:57:20.8204757Z Ran 1 test in 2.846s 2022-03-17T19:57:20.8204926Z 2022-03-17T19:57:20.8205000Z OK 2022-03-17T19:57:20.8205140Z 2022-03-17T19:57:20.8205272Z Generating XML reports... 2022-03-17T19:57:20.8248193Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195717.xml 2022-03-17T19:57:21.8826750Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:21.8840267Z 2022-03-17T19:57:21.8840412Z Running tests... 2022-03-17T19:57:21.8841797Z ---------------------------------------------------------------------- 2022-03-17T19:57:23.4359166Z test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:23.4800887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88409 2022-03-17T19:57:23.4930904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88410 2022-03-17T19:57:23.5061850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88411 2022-03-17T19:57:23.5195970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88412 2022-03-17T19:57:24.3627443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:24.4111211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:24.4220193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:24.4229841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:25.3250002Z ok (3.441s) 2022-03-17T19:57:25.3250246Z 2022-03-17T19:57:25.3250659Z ---------------------------------------------------------------------- 2022-03-17T19:57:25.3250928Z Ran 1 test in 3.441s 2022-03-17T19:57:25.3251094Z 2022-03-17T19:57:25.3251190Z OK 2022-03-17T19:57:25.3251325Z 2022-03-17T19:57:25.3251461Z Generating XML reports... 2022-03-17T19:57:25.3293929Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195721.xml 2022-03-17T19:57:26.3963168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:26.3976107Z 2022-03-17T19:57:26.3976566Z Running tests... 2022-03-17T19:57:26.3977076Z ---------------------------------------------------------------------- 2022-03-17T19:57:27.9261482Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:27.9681389Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88626 2022-03-17T19:57:27.9802885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88627 2022-03-17T19:57:27.9929200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88628 2022-03-17T19:57:28.0059307Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88629 2022-03-17T19:57:28.8607741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:28.8614371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:28.8757124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:28.8938887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:32.6157468Z ok (6.218s) 2022-03-17T19:57:32.6157713Z 2022-03-17T19:57:32.6158121Z ---------------------------------------------------------------------- 2022-03-17T19:57:32.6158466Z Ran 1 test in 6.218s 2022-03-17T19:57:32.6158613Z 2022-03-17T19:57:32.6159000Z OK 2022-03-17T19:57:32.6159154Z 2022-03-17T19:57:32.6159287Z Generating XML reports... 2022-03-17T19:57:32.6201772Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195726.xml 2022-03-17T19:57:33.6892195Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:33.6906581Z 2022-03-17T19:57:33.6906951Z Running tests... 2022-03-17T19:57:33.6907868Z ---------------------------------------------------------------------- 2022-03-17T19:57:35.2315314Z test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:35.2741796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88847 2022-03-17T19:57:35.2863433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88848 2022-03-17T19:57:35.2991798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88849 2022-03-17T19:57:35.3114873Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88850 2022-03-17T19:57:36.1631724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:36.1746888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:36.1827048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:36.2117786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:36.5158373Z ok (2.825s) 2022-03-17T19:57:36.5158601Z 2022-03-17T19:57:36.5158977Z ---------------------------------------------------------------------- 2022-03-17T19:57:36.5159323Z Ran 1 test in 2.825s 2022-03-17T19:57:36.5159490Z 2022-03-17T19:57:36.5159594Z OK 2022-03-17T19:57:36.5159705Z 2022-03-17T19:57:36.5159833Z Generating XML reports... 2022-03-17T19:57:36.5204216Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195733.xml 2022-03-17T19:57:37.5595327Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:37.5609601Z 2022-03-17T19:57:37.5609729Z Running tests... 2022-03-17T19:57:37.5610591Z ---------------------------------------------------------------------- 2022-03-17T19:57:39.0947825Z test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:39.1375380Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89044 2022-03-17T19:57:39.1497988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89045 2022-03-17T19:57:39.1623351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89046 2022-03-17T19:57:39.1755019Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89047 2022-03-17T19:57:40.0101348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:40.0553662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:40.0644761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:40.0653452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:40.3795769Z ok (2.818s) 2022-03-17T19:57:40.3795925Z 2022-03-17T19:57:40.3796483Z ---------------------------------------------------------------------- 2022-03-17T19:57:40.3797000Z Ran 1 test in 2.819s 2022-03-17T19:57:40.3797145Z 2022-03-17T19:57:40.3797240Z OK 2022-03-17T19:57:40.3797382Z 2022-03-17T19:57:40.3797494Z Generating XML reports... 2022-03-17T19:57:40.3840974Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195737.xml 2022-03-17T19:57:41.4310504Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:41.4324991Z 2022-03-17T19:57:41.4325252Z Running tests... 2022-03-17T19:57:41.4326104Z ---------------------------------------------------------------------- 2022-03-17T19:57:42.9778832Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:43.0211669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89237 2022-03-17T19:57:43.0336360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89238 2022-03-17T19:57:43.0466599Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89239 2022-03-17T19:57:43.0597378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89240 2022-03-17T19:57:43.9708916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:43.9744791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:43.9900403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:44.0177481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:45.9674801Z ok (4.535s) 2022-03-17T19:57:45.9675067Z 2022-03-17T19:57:45.9675519Z ---------------------------------------------------------------------- 2022-03-17T19:57:45.9675815Z Ran 1 test in 4.535s 2022-03-17T19:57:45.9675966Z 2022-03-17T19:57:45.9676033Z OK 2022-03-17T19:57:45.9676169Z 2022-03-17T19:57:45.9676301Z Generating XML reports... 2022-03-17T19:57:45.9719187Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195741.xml 2022-03-17T19:57:47.0350167Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:47.0364692Z 2022-03-17T19:57:47.0365246Z Running tests... 2022-03-17T19:57:47.0365697Z ---------------------------------------------------------------------- 2022-03-17T19:57:48.5629562Z test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:48.6056237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89434 2022-03-17T19:57:48.6178308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89435 2022-03-17T19:57:48.6305556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89436 2022-03-17T19:57:48.6434758Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89437 2022-03-17T19:57:49.4777050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:49.4781445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:49.4931495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:49.5064226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:49.7475356Z ok (2.711s) 2022-03-17T19:57:49.7475563Z 2022-03-17T19:57:49.7475980Z ---------------------------------------------------------------------- 2022-03-17T19:57:49.7476323Z Ran 1 test in 2.711s 2022-03-17T19:57:49.7476492Z 2022-03-17T19:57:49.7476566Z OK 2022-03-17T19:57:49.7477044Z 2022-03-17T19:57:49.7477181Z Generating XML reports... 2022-03-17T19:57:49.7520627Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195747.xml 2022-03-17T19:57:50.7983929Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:50.8006260Z 2022-03-17T19:57:50.8006760Z Running tests... 2022-03-17T19:57:50.8007232Z ---------------------------------------------------------------------- 2022-03-17T19:57:52.3349638Z test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:52.3777084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89627 2022-03-17T19:57:52.3899801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89628 2022-03-17T19:57:52.4025633Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89629 2022-03-17T19:57:52.4157414Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89630 2022-03-17T19:57:53.2186544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:53.2199623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:53.2723423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:53.2735233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:57:53.9206833Z ok (3.120s) 2022-03-17T19:57:53.9207064Z 2022-03-17T19:57:53.9207481Z ---------------------------------------------------------------------- 2022-03-17T19:57:53.9207821Z Ran 1 test in 3.120s 2022-03-17T19:57:53.9207988Z 2022-03-17T19:57:53.9208060Z OK 2022-03-17T19:57:53.9208199Z 2022-03-17T19:57:53.9208333Z Generating XML reports... 2022-03-17T19:57:53.9251227Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195750.xml 2022-03-17T19:57:54.9827863Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:57:54.9840971Z 2022-03-17T19:57:54.9841276Z Running tests... 2022-03-17T19:57:56.4946345Z ---------------------------------------------------------------------- 2022-03-17T19:57:56.4946864Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:57:56.5368267Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89844 2022-03-17T19:57:56.5490670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89845 2022-03-17T19:57:56.5615239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89846 2022-03-17T19:57:56.5743650Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89847 2022-03-17T19:57:57.4331737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:57:57.4526636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:57:57.4670571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:57:57.4786893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:00.2834818Z ok (5.299s) 2022-03-17T19:58:00.2835111Z 2022-03-17T19:58:00.2835460Z ---------------------------------------------------------------------- 2022-03-17T19:58:00.2835793Z Ran 1 test in 5.299s 2022-03-17T19:58:00.2835947Z 2022-03-17T19:58:00.2836037Z OK 2022-03-17T19:58:00.2836171Z 2022-03-17T19:58:00.2836318Z Generating XML reports... 2022-03-17T19:58:00.2879511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195754.xml 2022-03-17T19:58:01.3495070Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:01.3509628Z 2022-03-17T19:58:01.3510055Z Running tests... 2022-03-17T19:58:01.3510657Z ---------------------------------------------------------------------- 2022-03-17T19:58:02.8944046Z test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:02.9373563Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90065 2022-03-17T19:58:02.9495008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90066 2022-03-17T19:58:02.9622034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90067 2022-03-17T19:58:02.9749355Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90068 2022-03-17T19:58:03.8156406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:03.8201718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:03.8265840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:03.8426130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:04.1794767Z ok (2.828s) 2022-03-17T19:58:04.1795152Z 2022-03-17T19:58:04.1795835Z ---------------------------------------------------------------------- 2022-03-17T19:58:04.1796382Z Ran 1 test in 2.828s 2022-03-17T19:58:04.1796621Z 2022-03-17T19:58:04.1796758Z OK 2022-03-17T19:58:04.1797000Z 2022-03-17T19:58:04.1797226Z Generating XML reports... 2022-03-17T19:58:04.1841786Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195801.xml 2022-03-17T19:58:05.2281660Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:05.2295322Z 2022-03-17T19:58:05.2295869Z Running tests... 2022-03-17T19:58:05.2296351Z ---------------------------------------------------------------------- 2022-03-17T19:58:06.7642347Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:06.8070826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90270 2022-03-17T19:58:06.8194227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90271 2022-03-17T19:58:06.8322508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90272 2022-03-17T19:58:06.8454459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90273 2022-03-17T19:58:07.6656550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:07.6871974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:07.6986559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:07.7618790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:08.2498993Z ok (3.020s) 2022-03-17T19:58:08.2499173Z 2022-03-17T19:58:08.2499591Z ---------------------------------------------------------------------- 2022-03-17T19:58:08.2499947Z Ran 1 test in 3.020s 2022-03-17T19:58:08.2500111Z 2022-03-17T19:58:08.2500201Z OK 2022-03-17T19:58:08.2500583Z 2022-03-17T19:58:08.2500715Z Generating XML reports... 2022-03-17T19:58:08.2543013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195805.xml 2022-03-17T19:58:09.3051993Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:09.3068517Z 2022-03-17T19:58:09.3068786Z Running tests... 2022-03-17T19:58:09.3069310Z ---------------------------------------------------------------------- 2022-03-17T19:58:10.8523704Z test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:10.8956464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90499 2022-03-17T19:58:10.9080645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90500 2022-03-17T19:58:10.9210558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90501 2022-03-17T19:58:10.9340091Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90502 2022-03-17T19:58:11.7159912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:11.7576686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:11.7846853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:11.8015282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:12.0380701Z ok (2.731s) 2022-03-17T19:58:12.0381482Z 2022-03-17T19:58:12.0382148Z ---------------------------------------------------------------------- 2022-03-17T19:58:12.0382493Z Ran 1 test in 2.731s 2022-03-17T19:58:12.0382672Z 2022-03-17T19:58:12.0382763Z OK 2022-03-17T19:58:12.0382898Z 2022-03-17T19:58:12.0383037Z Generating XML reports... 2022-03-17T19:58:12.0425671Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195809.xml 2022-03-17T19:58:13.0879421Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:13.0893298Z 2022-03-17T19:58:13.0893536Z Running tests... 2022-03-17T19:58:13.0894337Z ---------------------------------------------------------------------- 2022-03-17T19:58:14.6131221Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:14.6560003Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90692 2022-03-17T19:58:14.6683535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90693 2022-03-17T19:58:14.6810465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90694 2022-03-17T19:58:14.6939446Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90695 2022-03-17T19:58:15.5346971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:15.5450760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:15.5863387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:15.5924086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:17.5015207Z ok (4.412s) 2022-03-17T19:58:17.5015422Z 2022-03-17T19:58:17.5015851Z ---------------------------------------------------------------------- 2022-03-17T19:58:17.5016197Z Ran 1 test in 4.412s 2022-03-17T19:58:17.5016367Z 2022-03-17T19:58:17.5016441Z OK 2022-03-17T19:58:17.5016581Z 2022-03-17T19:58:17.5016720Z Generating XML reports... 2022-03-17T19:58:17.5059422Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195813.xml 2022-03-17T19:58:18.5646084Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:18.5660718Z 2022-03-17T19:58:18.5660974Z Running tests... 2022-03-17T19:58:18.5661404Z ---------------------------------------------------------------------- 2022-03-17T19:58:20.0979258Z test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:20.1406778Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90889 2022-03-17T19:58:20.1530561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90890 2022-03-17T19:58:20.1656088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90891 2022-03-17T19:58:20.1787875Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90892 2022-03-17T19:58:20.9974331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:21.0086332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:21.0213067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:21.0978413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:21.3829538Z ok (2.817s) 2022-03-17T19:58:21.3829747Z 2022-03-17T19:58:21.3830175Z ---------------------------------------------------------------------- 2022-03-17T19:58:21.3830485Z Ran 1 test in 2.817s 2022-03-17T19:58:21.3830654Z 2022-03-17T19:58:21.3830727Z OK 2022-03-17T19:58:21.3830860Z 2022-03-17T19:58:21.3831246Z Generating XML reports... 2022-03-17T19:58:21.3872767Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195818.xml 2022-03-17T19:58:22.4285145Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:22.4298624Z 2022-03-17T19:58:22.4298792Z Running tests... 2022-03-17T19:58:22.4299477Z ---------------------------------------------------------------------- 2022-03-17T19:58:23.9662553Z test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:24.0091871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91082 2022-03-17T19:58:24.0215408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91083 2022-03-17T19:58:24.0343990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91084 2022-03-17T19:58:24.0475873Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91085 2022-03-17T19:58:24.8353427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:24.8521225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:24.9294171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:24.9332136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:25.8529149Z ok (3.423s) 2022-03-17T19:58:25.8529384Z 2022-03-17T19:58:25.8529784Z ---------------------------------------------------------------------- 2022-03-17T19:58:25.8530105Z Ran 1 test in 3.423s 2022-03-17T19:58:25.8530271Z 2022-03-17T19:58:25.8530361Z OK 2022-03-17T19:58:25.8530497Z 2022-03-17T19:58:25.8530618Z Generating XML reports... 2022-03-17T19:58:25.8574068Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195822.xml 2022-03-17T19:58:26.9219353Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:26.9233138Z 2022-03-17T19:58:26.9233302Z Running tests... 2022-03-17T19:58:26.9234577Z ---------------------------------------------------------------------- 2022-03-17T19:58:26.9241818Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip (0.001s) 2022-03-17T19:58:26.9242087Z 2022-03-17T19:58:26.9242409Z ---------------------------------------------------------------------- 2022-03-17T19:58:26.9242951Z Ran 1 test in 0.001s 2022-03-17T19:58:26.9243100Z 2022-03-17T19:58:26.9243205Z OK (skipped=1) 2022-03-17T19:58:26.9243326Z 2022-03-17T19:58:26.9243437Z Generating XML reports... 2022-03-17T19:58:26.9277976Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195826.xml 2022-03-17T19:58:27.8280964Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:27.8294338Z 2022-03-17T19:58:27.8294626Z Running tests... 2022-03-17T19:58:27.8295254Z ---------------------------------------------------------------------- 2022-03-17T19:58:29.3849527Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:29.4281233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91334 2022-03-17T19:58:29.4407956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91335 2022-03-17T19:58:29.4533553Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91336 2022-03-17T19:58:29.4664902Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91337 2022-03-17T19:58:30.3298789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:30.3412834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:30.3641227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:30.4137969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:30.6708119Z ok (2.841s) 2022-03-17T19:58:30.6708519Z 2022-03-17T19:58:30.6709020Z ---------------------------------------------------------------------- 2022-03-17T19:58:30.6709327Z Ran 1 test in 2.841s 2022-03-17T19:58:30.6709488Z 2022-03-17T19:58:30.6709590Z OK 2022-03-17T19:58:30.6709728Z 2022-03-17T19:58:30.6709854Z Generating XML reports... 2022-03-17T19:58:30.6752015Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195827.xml 2022-03-17T19:58:31.7176149Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:31.7190334Z 2022-03-17T19:58:31.7190475Z Running tests... 2022-03-17T19:58:31.7191199Z ---------------------------------------------------------------------- 2022-03-17T19:58:31.7195636Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip (0.000s) 2022-03-17T19:58:31.7195904Z 2022-03-17T19:58:31.7196234Z ---------------------------------------------------------------------- 2022-03-17T19:58:31.7196554Z Ran 1 test in 0.001s 2022-03-17T19:58:31.7196720Z 2022-03-17T19:58:31.7196808Z OK (skipped=1) 2022-03-17T19:58:31.7196958Z 2022-03-17T19:58:31.7197068Z Generating XML reports... 2022-03-17T19:58:31.7231527Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195831.xml 2022-03-17T19:58:32.6264416Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:32.6278033Z 2022-03-17T19:58:32.6278494Z Running tests... 2022-03-17T19:58:32.6278993Z ---------------------------------------------------------------------- 2022-03-17T19:58:34.1574804Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:34.2003320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91562 2022-03-17T19:58:34.2125255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91563 2022-03-17T19:58:34.2252008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91564 2022-03-17T19:58:34.2383396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91565 2022-03-17T19:58:35.0255980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:35.0301206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:35.1209628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:35.1255136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:37.2460119Z ok (4.618s) 2022-03-17T19:58:37.2460633Z 2022-03-17T19:58:37.2461060Z ---------------------------------------------------------------------- 2022-03-17T19:58:37.2461384Z Ran 1 test in 4.618s 2022-03-17T19:58:37.2461548Z 2022-03-17T19:58:37.2461641Z OK 2022-03-17T19:58:37.2463810Z 2022-03-17T19:58:37.2464353Z Generating XML reports... 2022-03-17T19:58:37.2504578Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195832.xml 2022-03-17T19:58:38.3132694Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:38.3146565Z 2022-03-17T19:58:38.3146663Z Running tests... 2022-03-17T19:58:38.3147618Z ---------------------------------------------------------------------- 2022-03-17T19:58:39.8350032Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:39.8776325Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91939 2022-03-17T19:58:39.8901346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91940 2022-03-17T19:58:39.9027053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91941 2022-03-17T19:58:39.9156654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91942 2022-03-17T19:58:40.7945709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T19:58:40.8042140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:40.8126177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3 2022-03-17T19:58:40.8157982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:58:41.1199079Z ok (2.805s) 2022-03-17T19:58:41.1199319Z 2022-03-17T19:58:41.1199693Z ---------------------------------------------------------------------- 2022-03-17T19:58:41.1200033Z Ran 1 test in 2.805s 2022-03-17T19:58:41.1200199Z 2022-03-17T19:58:41.1200294Z OK 2022-03-17T19:58:41.1200412Z 2022-03-17T19:58:41.1200561Z Generating XML reports... 2022-03-17T19:58:41.1244336Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195838.xml 2022-03-17T19:58:42.1690764Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:42.1704954Z 2022-03-17T19:58:42.1705465Z Running tests... 2022-03-17T19:58:42.1705942Z ---------------------------------------------------------------------- 2022-03-17T19:58:42.1778989Z test_forward_backward (__main__.ReducerTest) ... ok (0.007s) 2022-03-17T19:58:42.1825011Z 2022-03-17T19:58:42.1825550Z ---------------------------------------------------------------------- 2022-03-17T19:58:42.1825896Z Ran 1 test in 0.012s 2022-03-17T19:58:42.1826059Z 2022-03-17T19:58:42.1826154Z OK 2022-03-17T19:58:42.1826290Z 2022-03-17T19:58:42.1826397Z Generating XML reports... 2022-03-17T19:58:42.1860315Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195842.xml 2022-03-17T19:58:43.0921553Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:43.0935892Z 2022-03-17T19:58:43.0936430Z Running tests... 2022-03-17T19:58:43.0936923Z ---------------------------------------------------------------------- 2022-03-17T19:58:43.1027194Z test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T19:58:43.1046927Z ok (0.011s) 2022-03-17T19:58:43.1059575Z 2022-03-17T19:58:43.1060000Z ---------------------------------------------------------------------- 2022-03-17T19:58:43.1060661Z Ran 1 test in 0.012s 2022-03-17T19:58:43.1060845Z 2022-03-17T19:58:43.1060938Z OK 2022-03-17T19:58:43.1061078Z 2022-03-17T19:58:43.1061186Z Generating XML reports... 2022-03-17T19:58:43.1096890Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195843.xml 2022-03-17T19:58:44.0140599Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:44.0155435Z 2022-03-17T19:58:44.0155729Z Running tests... 2022-03-17T19:58:44.0156181Z ---------------------------------------------------------------------- 2022-03-17T19:58:44.0233093Z test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.008s) 2022-03-17T19:58:44.0280807Z 2022-03-17T19:58:44.0281459Z ---------------------------------------------------------------------- 2022-03-17T19:58:44.0281774Z Ran 1 test in 0.012s 2022-03-17T19:58:44.0281943Z 2022-03-17T19:58:44.0282296Z OK 2022-03-17T19:58:44.0282412Z 2022-03-17T19:58:44.0282538Z Generating XML reports... 2022-03-17T19:58:44.0317019Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195844.xml 2022-03-17T19:58:44.9327155Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:44.9341065Z 2022-03-17T19:58:44.9341481Z Running tests... 2022-03-17T19:58:44.9342000Z ---------------------------------------------------------------------- 2022-03-17T19:58:44.9383975Z test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.004s) 2022-03-17T19:58:44.9460606Z 2022-03-17T19:58:44.9461023Z ---------------------------------------------------------------------- 2022-03-17T19:58:44.9461543Z Ran 1 test in 0.012s 2022-03-17T19:58:44.9461837Z 2022-03-17T19:58:44.9461992Z OK 2022-03-17T19:58:44.9462232Z 2022-03-17T19:58:44.9462397Z Generating XML reports... 2022-03-17T19:58:44.9504385Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195844.xml 2022-03-17T19:58:45.8292462Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:45.8306679Z 2022-03-17T19:58:45.8306952Z Running tests... 2022-03-17T19:58:45.8307433Z ---------------------------------------------------------------------- 2022-03-17T19:58:45.8374100Z test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.007s) 2022-03-17T19:58:45.8425061Z 2022-03-17T19:58:45.8425546Z ---------------------------------------------------------------------- 2022-03-17T19:58:45.8425839Z Ran 1 test in 0.012s 2022-03-17T19:58:45.8426004Z 2022-03-17T19:58:45.8426068Z OK 2022-03-17T19:58:45.8426210Z 2022-03-17T19:58:45.8426338Z Generating XML reports... 2022-03-17T19:58:45.8461991Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195845.xml 2022-03-17T19:58:46.7523186Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:46.7537949Z 2022-03-17T19:58:46.7538109Z Running tests... 2022-03-17T19:58:46.7538801Z ---------------------------------------------------------------------- 2022-03-17T19:58:46.7574744Z test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.004s) 2022-03-17T19:58:46.7656883Z 2022-03-17T19:58:46.7657147Z ---------------------------------------------------------------------- 2022-03-17T19:58:46.7657780Z Ran 1 test in 0.012s 2022-03-17T19:58:46.7657970Z 2022-03-17T19:58:46.7658053Z OK 2022-03-17T19:58:46.7658178Z 2022-03-17T19:58:46.7658297Z Generating XML reports... 2022-03-17T19:58:46.7694007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195846.xml 2022-03-17T19:58:47.6702776Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:47.6717246Z 2022-03-17T19:58:47.6717635Z Running tests... 2022-03-17T19:58:47.6718205Z ---------------------------------------------------------------------- 2022-03-17T19:58:49.1941986Z test_logging_init (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:49.2113310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:58:49.2114293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:58:49.2211136Z ok (1.549s) 2022-03-17T19:58:49.2211551Z 2022-03-17T19:58:49.2211847Z ---------------------------------------------------------------------- 2022-03-17T19:58:49.2212529Z Ran 1 test in 1.549s 2022-03-17T19:58:49.2212964Z 2022-03-17T19:58:49.2213052Z OK 2022-03-17T19:58:49.2213287Z 2022-03-17T19:58:49.2213426Z Generating XML reports... 2022-03-17T19:58:49.2246390Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220317195847.xml 2022-03-17T19:58:50.2368332Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo 2022-03-17T19:58:50.2382395Z 2022-03-17T19:58:50.2382530Z Running tests... 2022-03-17T19:58:50.2383249Z ---------------------------------------------------------------------- 2022-03-17T19:58:51.7724295Z test_default_store_timeout_gloo (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:51.7899933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:58:51.7900927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:58:53.8007964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:58:53.8008780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T19:58:54.8206320Z ok (4.582s) 2022-03-17T19:58:54.8206910Z 2022-03-17T19:58:54.8207797Z ---------------------------------------------------------------------- 2022-03-17T19:58:54.8208215Z Ran 1 test in 4.582s 2022-03-17T19:58:54.8208397Z 2022-03-17T19:58:54.8208500Z OK 2022-03-17T19:58:54.8208638Z 2022-03-17T19:58:54.8208768Z Generating XML reports... 2022-03-17T19:58:54.8251921Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220317195850.xml 2022-03-17T19:58:55.1643389Z Running distributed/test_c10d_nccl ... [2022-03-17 19:58:55.163774] 2022-03-17T19:58:55.1644075Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 19:58:55.163883] 2022-03-17T19:58:55.9973503Z , <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>]> 2022-03-17T19:58:55.9975787Z test_all_reduce_coalesced_nccl (__main__.CommTest) 2022-03-17T19:58:55.9976480Z test_broadcast_coalesced_nccl (__main__.CommTest) 2022-03-17T19:58:55.9976985Z test_nccl_barrier (__main__.CommTest) 2022-03-17T19:58:55.9977501Z test_nccl_barrier_device_ids (__main__.CommTest) 2022-03-17T19:58:55.9978233Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) 2022-03-17T19:58:55.9978933Z test_nccl_barrier_timeout (__main__.CommTest) 2022-03-17T19:58:55.9979697Z test_nccl_barrier_timeout_new_group (__main__.CommTest) 2022-03-17T19:58:55.9980163Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) 2022-03-17T19:58:55.9981194Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) 2022-03-17T19:58:55.9981854Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) 2022-03-17T19:58:55.9982433Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) 2022-03-17T19:58:55.9983093Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) 2022-03-17T19:58:55.9983489Z test_sequence_num_incremented_nccl_default (__main__.CommTest) 2022-03-17T19:58:55.9984105Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) 2022-03-17T19:58:55.9984456Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) 2022-03-17T19:58:55.9984987Z test_sequence_num_set_nccl_new_group (__main__.CommTest) 2022-03-17T19:58:55.9995546Z , <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>]> 2022-03-17T19:58:56.0004274Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0004733Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0005198Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0005638Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0006093Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0006530Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0006940Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0007384Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0007830Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0008299Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0008777Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0009236Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0009733Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0010225Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0010792Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0011269Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0011723Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0012213Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0012695Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0013308Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0013819Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0014297Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0014744Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0015222Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0015676Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0016138Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0016585Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0016991Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0017411Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0017823Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0018265Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0018685Z test_failure_recovery (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0019106Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0019600Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0020887Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0021707Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0022645Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0023544Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0024336Z test_fp16 (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0025040Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0025849Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0026600Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0027344Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0028145Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0028666Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0029103Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0029581Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0030049Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0030544Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0030986Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0031409Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0031980Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0032435Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0032918Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0033388Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0033852Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0034255Z test_no_grad (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0034706Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0035175Z test_pass_default_pg (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0035571Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0036024Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) 2022-03-17T19:58:56.0036414Z 2022-03-17T19:58:56.0037667Z , <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]> 2022-03-17T19:58:56.0038939Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0039333Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0039734Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0040115Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0040520Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0040924Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0041300Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0041691Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0042039Z test_nccl_timeout (__main__.NcclErrorHandlingTest) 2022-03-17T19:58:56.0042488Z ]> 2022-03-17T19:58:56.0042945Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) 2022-03-17T19:58:56.0044729Z , <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>]> 2022-03-17T19:58:56.0046702Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0047066Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0047431Z test_allgather_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0047787Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0048114Z test_barrier (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0048462Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0048799Z test_empty_tensors (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0049149Z test_gather_checks (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0049534Z test_gather_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0049894Z test_gather_stress (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0050241Z test_reduce_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0050594Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0050989Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0051346Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0051707Z test_scatter_checks (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0052057Z test_scatter_ops (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0052391Z test_scatter_stress (__main__.ProcessGroupNCCLTest) 2022-03-17T19:58:56.0052815Z ]> 2022-03-17T19:58:56.0053207Z test_common_errors (__main__.RendezvousEnvTest) 2022-03-17T19:58:56.0053530Z 2022-03-17T19:58:56.0053928Z ]> 2022-03-17T19:58:56.0054352Z test_default_store_timeout_nccl (__main__.TimeoutTest) 2022-03-17T19:58:56.8076979Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:58:56.8091949Z 2022-03-17T19:58:56.8092169Z Running tests... 2022-03-17T19:58:56.8093054Z ---------------------------------------------------------------------- 2022-03-17T19:58:58.3539606Z test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:58:58.3964492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92533 2022-03-17T19:58:58.4087134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92534 2022-03-17T19:58:59.2135466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:58:59.2755728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:01.9167025Z ok (5.107s) 2022-03-17T19:59:01.9167418Z 2022-03-17T19:59:01.9168215Z ---------------------------------------------------------------------- 2022-03-17T19:59:01.9168727Z Ran 1 test in 5.107s 2022-03-17T19:59:01.9168892Z 2022-03-17T19:59:01.9168984Z OK 2022-03-17T19:59:01.9169122Z 2022-03-17T19:59:01.9169255Z Generating XML reports... 2022-03-17T19:59:01.9210953Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195856.xml 2022-03-17T19:59:02.9879143Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:02.9894239Z 2022-03-17T19:59:02.9894392Z Running tests... 2022-03-17T19:59:02.9895113Z ---------------------------------------------------------------------- 2022-03-17T19:59:04.5450393Z test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:04.5876276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92656 2022-03-17T19:59:04.6002384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92657 2022-03-17T19:59:05.4012972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:05.4013787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:08.1085537Z ok (5.119s) 2022-03-17T19:59:08.1085778Z 2022-03-17T19:59:08.1086404Z ---------------------------------------------------------------------- 2022-03-17T19:59:08.1086772Z Ran 1 test in 5.119s 2022-03-17T19:59:08.1086945Z 2022-03-17T19:59:08.1087029Z OK 2022-03-17T19:59:08.1087171Z 2022-03-17T19:59:08.1087308Z Generating XML reports... 2022-03-17T19:59:08.1130278Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195902.xml 2022-03-17T19:59:09.1709757Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:09.1725666Z 2022-03-17T19:59:09.1725838Z Running tests... 2022-03-17T19:59:09.1726327Z ---------------------------------------------------------------------- 2022-03-17T19:59:10.7056252Z test_nccl_barrier (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:10.7482706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92779 2022-03-17T19:59:10.7607010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92780 2022-03-17T19:59:11.5902612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:11.6234401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:11.7643711Z skip (2.591s) 2022-03-17T19:59:11.7643933Z 2022-03-17T19:59:11.7644318Z ---------------------------------------------------------------------- 2022-03-17T19:59:11.7644642Z Ran 1 test in 2.592s 2022-03-17T19:59:11.7644806Z 2022-03-17T19:59:11.7644938Z OK (skipped=1) 2022-03-17T19:59:11.7645084Z 2022-03-17T19:59:11.7645205Z Generating XML reports... 2022-03-17T19:59:11.7687495Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195909.xml 2022-03-17T19:59:12.8088334Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:12.8103324Z 2022-03-17T19:59:12.8103610Z Running tests... 2022-03-17T19:59:12.8104066Z ---------------------------------------------------------------------- 2022-03-17T19:59:14.3405968Z test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:14.3842823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92888 2022-03-17T19:59:14.3966561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92889 2022-03-17T19:59:15.2218106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:15.2220863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:59:15.2365628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:15.2369896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:59:15.2370901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:15.2424839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:16.9030031Z ok (4.092s) 2022-03-17T19:59:16.9030241Z 2022-03-17T19:59:16.9030654Z ---------------------------------------------------------------------- 2022-03-17T19:59:16.9030995Z Ran 1 test in 4.093s 2022-03-17T19:59:16.9031129Z 2022-03-17T19:59:16.9031200Z OK 2022-03-17T19:59:16.9031337Z 2022-03-17T19:59:16.9031473Z Generating XML reports... 2022-03-17T19:59:16.9074174Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195912.xml 2022-03-17T19:59:17.9613729Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:17.9628244Z 2022-03-17T19:59:17.9628662Z Running tests... 2022-03-17T19:59:17.9629132Z ---------------------------------------------------------------------- 2022-03-17T19:59:19.4686831Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:19.5110967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93010 2022-03-17T19:59:19.5233875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93011 2022-03-17T19:59:20.3261284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:20.3264179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:59:20.3741105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:20.3745480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:59:20.3746281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:20.3771473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:20.5269961Z ok (2.564s) 2022-03-17T19:59:20.5270180Z 2022-03-17T19:59:20.5270579Z ---------------------------------------------------------------------- 2022-03-17T19:59:20.5270886Z Ran 1 test in 2.564s 2022-03-17T19:59:20.5271065Z 2022-03-17T19:59:20.5271163Z OK 2022-03-17T19:59:20.5271291Z 2022-03-17T19:59:20.5271418Z Generating XML reports... 2022-03-17T19:59:20.5313978Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195917.xml 2022-03-17T19:59:21.5741006Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:21.5756334Z 2022-03-17T19:59:21.5756662Z Running tests... 2022-03-17T19:59:21.5757090Z ---------------------------------------------------------------------- 2022-03-17T19:59:23.1114322Z test_nccl_barrier_timeout (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:23.1541938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93123 2022-03-17T19:59:23.1666657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93124 2022-03-17T19:59:23.9726064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:23.9782199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:24.1701775Z skip (2.594s) 2022-03-17T19:59:24.1702173Z 2022-03-17T19:59:24.1702928Z ---------------------------------------------------------------------- 2022-03-17T19:59:24.1703403Z Ran 1 test in 2.594s 2022-03-17T19:59:24.1703573Z 2022-03-17T19:59:24.1703686Z OK (skipped=1) 2022-03-17T19:59:24.1703823Z 2022-03-17T19:59:24.1703949Z Generating XML reports... 2022-03-17T19:59:24.1746001Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195921.xml 2022-03-17T19:59:25.2155319Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:25.2171127Z 2022-03-17T19:59:25.2171696Z Running tests... 2022-03-17T19:59:25.2172177Z ---------------------------------------------------------------------- 2022-03-17T19:59:26.7508468Z test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:26.7936232Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93232 2022-03-17T19:59:26.8060776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93233 2022-03-17T19:59:27.6513518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:27.6907798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:27.8095844Z skip (2.592s) 2022-03-17T19:59:27.8096102Z 2022-03-17T19:59:27.8096503Z ---------------------------------------------------------------------- 2022-03-17T19:59:27.8096849Z Ran 1 test in 2.592s 2022-03-17T19:59:27.8097022Z 2022-03-17T19:59:27.8097096Z OK (skipped=1) 2022-03-17T19:59:27.8097253Z 2022-03-17T19:59:27.8097381Z Generating XML reports... 2022-03-17T19:59:27.8140857Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195925.xml 2022-03-17T19:59:28.8569845Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:28.8585590Z 2022-03-17T19:59:28.8586016Z Running tests... 2022-03-17T19:59:28.8586761Z ---------------------------------------------------------------------- 2022-03-17T19:59:30.3895552Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:30.4325584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93341 2022-03-17T19:59:30.4448758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93342 2022-03-17T19:59:31.2793361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:31.2794165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:31.4484515Z skip (2.589s) 2022-03-17T19:59:31.4484913Z 2022-03-17T19:59:31.4485499Z ---------------------------------------------------------------------- 2022-03-17T19:59:31.4485813Z Ran 1 test in 2.590s 2022-03-17T19:59:31.4485977Z 2022-03-17T19:59:31.4486069Z OK (skipped=1) 2022-03-17T19:59:31.4486223Z 2022-03-17T19:59:31.4486362Z Generating XML reports... 2022-03-17T19:59:31.4528831Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195928.xml 2022-03-17T19:59:32.5023932Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:32.5040771Z 2022-03-17T19:59:32.5041304Z Running tests... 2022-03-17T19:59:32.5041805Z ---------------------------------------------------------------------- 2022-03-17T19:59:34.0358331Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:34.0787187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93450 2022-03-17T19:59:34.0912429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93451 2022-03-17T19:59:34.8998735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:34.8999619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:34.9211821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:59:34.9212396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:59:34.9213197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:34.9213897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:34.9214419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:59:34.9218919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:59:34.9219602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:34.9317715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:36.6977700Z ok (4.193s) 2022-03-17T19:59:36.6978036Z 2022-03-17T19:59:36.6978838Z ---------------------------------------------------------------------- 2022-03-17T19:59:36.6979560Z Ran 1 test in 4.194s 2022-03-17T19:59:36.6979705Z 2022-03-17T19:59:36.6979801Z OK 2022-03-17T19:59:36.6979935Z 2022-03-17T19:59:36.6980070Z Generating XML reports... 2022-03-17T19:59:36.7021920Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195932.xml 2022-03-17T19:59:37.6982340Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:37.6999400Z 2022-03-17T19:59:37.6999902Z Running tests... 2022-03-17T19:59:37.7000399Z ---------------------------------------------------------------------- 2022-03-17T19:59:39.2419956Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:39.2846218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93587 2022-03-17T19:59:39.2970143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93588 2022-03-17T19:59:40.1181135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:40.1184582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:59:40.1309075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:40.1314082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:59:40.1314862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:40.1315856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:59:40.1390780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:40.1393334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:59:40.1394176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:40.1418777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:41.8033271Z ok (4.103s) 2022-03-17T19:59:41.8033657Z 2022-03-17T19:59:41.8034227Z ---------------------------------------------------------------------- 2022-03-17T19:59:41.8034817Z Ran 1 test in 4.103s 2022-03-17T19:59:41.8035120Z 2022-03-17T19:59:41.8035277Z OK 2022-03-17T19:59:41.8035522Z 2022-03-17T19:59:41.8035841Z Generating XML reports... 2022-03-17T19:59:41.8079316Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195937.xml 2022-03-17T19:59:42.8712635Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:42.8727404Z 2022-03-17T19:59:42.8727521Z Running tests... 2022-03-17T19:59:42.8728272Z ---------------------------------------------------------------------- 2022-03-17T19:59:44.4127511Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:44.4556070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93715 2022-03-17T19:59:44.4680048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93716 2022-03-17T19:59:45.2710031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:45.2712657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:59:45.3085965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:45.3089906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:59:45.3091086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:45.3092456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:59:45.3119517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:45.3122108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:59:45.3123088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:45.3195681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:46.9745645Z ok (4.101s) 2022-03-17T19:59:46.9745918Z 2022-03-17T19:59:46.9746307Z ---------------------------------------------------------------------- 2022-03-17T19:59:46.9746643Z Ran 1 test in 4.102s 2022-03-17T19:59:46.9746785Z 2022-03-17T19:59:46.9746904Z OK 2022-03-17T19:59:46.9747005Z 2022-03-17T19:59:46.9747145Z Generating XML reports... 2022-03-17T19:59:46.9790327Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195942.xml 2022-03-17T19:59:48.0449463Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:48.0464680Z 2022-03-17T19:59:48.0464945Z Running tests... 2022-03-17T19:59:48.0465355Z ---------------------------------------------------------------------- 2022-03-17T19:59:49.5829746Z test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:49.6260612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93843 2022-03-17T19:59:49.6384018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93844 2022-03-17T19:59:50.4514032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:50.4516118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:59:50.4627353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:50.4631489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:59:50.4632680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:50.4635452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:59:50.4720395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:50.4722427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:59:50.4723634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:50.4738266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:53.1468361Z ok (5.100s) 2022-03-17T19:59:53.1471712Z 2022-03-17T19:59:53.1472643Z ---------------------------------------------------------------------- 2022-03-17T19:59:53.1473379Z Ran 1 test in 5.100s 2022-03-17T19:59:53.1473620Z 2022-03-17T19:59:53.1473767Z OK 2022-03-17T19:59:53.1473910Z 2022-03-17T19:59:53.1474067Z Generating XML reports... 2022-03-17T19:59:53.1518245Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195948.xml 2022-03-17T19:59:54.2119976Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:54.2135797Z 2022-03-17T19:59:54.2136021Z Running tests... 2022-03-17T19:59:54.2136462Z ---------------------------------------------------------------------- 2022-03-17T19:59:55.7571881Z test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T19:59:55.8004284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93970 2022-03-17T19:59:55.8128333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93971 2022-03-17T19:59:56.6448305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T19:59:56.6449119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T19:59:56.6467728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T19:59:56.6468973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T19:59:56.6470420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:56.6471769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T19:59:56.6578235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T19:59:56.6579067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T19:59:56.6580632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:56.6684274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T19:59:58.3190862Z ok (4.105s) 2022-03-17T19:59:58.3191148Z 2022-03-17T19:59:58.3191534Z ---------------------------------------------------------------------- 2022-03-17T19:59:58.3191856Z Ran 1 test in 4.105s 2022-03-17T19:59:58.3192019Z 2022-03-17T19:59:58.3192103Z OK 2022-03-17T19:59:58.3192219Z 2022-03-17T19:59:58.3192334Z Generating XML reports... 2022-03-17T19:59:58.3235918Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195954.xml 2022-03-17T19:59:59.3881900Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T19:59:59.3897513Z 2022-03-17T19:59:59.3897784Z Running tests... 2022-03-17T19:59:59.3898153Z ---------------------------------------------------------------------- 2022-03-17T20:00:00.9290171Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:00.9722401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94098 2022-03-17T20:00:00.9849083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94099 2022-03-17T20:00:01.8231682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:01.8654615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:01.9884511Z skip (2.598s) 2022-03-17T20:00:01.9884682Z 2022-03-17T20:00:01.9885111Z ---------------------------------------------------------------------- 2022-03-17T20:00:01.9885441Z Ran 1 test in 2.599s 2022-03-17T20:00:01.9885607Z 2022-03-17T20:00:01.9885648Z OK (skipped=1) 2022-03-17T20:00:01.9885797Z 2022-03-17T20:00:01.9885926Z Generating XML reports... 2022-03-17T20:00:01.9929403Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195959.xml 2022-03-17T20:00:03.0533007Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:03.0548942Z 2022-03-17T20:00:03.0549476Z Running tests... 2022-03-17T20:00:03.0549919Z ---------------------------------------------------------------------- 2022-03-17T20:00:04.5785997Z test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:04.6214843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94207 2022-03-17T20:00:04.6340196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94208 2022-03-17T20:00:05.4725413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:05.4735938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:00:05.5137722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:05.5152929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:00:05.5154192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:00:05.5244044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:00:07.1402945Z ok (4.085s) 2022-03-17T20:00:07.1403155Z 2022-03-17T20:00:07.1403546Z ---------------------------------------------------------------------- 2022-03-17T20:00:07.1403882Z Ran 1 test in 4.085s 2022-03-17T20:00:07.1404049Z 2022-03-17T20:00:07.1404140Z OK 2022-03-17T20:00:07.1404276Z 2022-03-17T20:00:07.1404414Z Generating XML reports... 2022-03-17T20:00:07.1447422Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317200003.xml 2022-03-17T20:00:08.2077614Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:08.2092245Z 2022-03-17T20:00:08.2092668Z Running tests... 2022-03-17T20:00:08.2093148Z ---------------------------------------------------------------------- 2022-03-17T20:00:09.7425013Z test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:09.7858300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94329 2022-03-17T20:00:09.7982033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94330 2022-03-17T20:00:10.6080694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:10.6083249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:10.6091229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:00:10.6095417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:00:10.6096265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:00:10.6098861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:00:10.6194870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:00:10.6197163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:00:10.6197844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:00:10.6201392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:00:12.3043855Z ok (4.095s) 2022-03-17T20:00:12.3044056Z 2022-03-17T20:00:12.3044443Z ---------------------------------------------------------------------- 2022-03-17T20:00:12.3044785Z Ran 1 test in 4.095s 2022-03-17T20:00:12.3044950Z 2022-03-17T20:00:12.3045029Z OK 2022-03-17T20:00:12.3045161Z 2022-03-17T20:00:12.3045310Z Generating XML reports... 2022-03-17T20:00:12.3086984Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317200008.xml 2022-03-17T20:00:13.3624450Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:13.3639582Z 2022-03-17T20:00:13.3639997Z Running tests... 2022-03-17T20:00:13.3640514Z ---------------------------------------------------------------------- 2022-03-17T20:00:14.9087082Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:14.9522213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94455 2022-03-17T20:00:14.9650862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94456 2022-03-17T20:00:15.7765032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:15.7805937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:18.4489842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:00:18.4490368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:00:18.7737695Z ok (5.410s) 2022-03-17T20:00:18.7737853Z 2022-03-17T20:00:18.7738280Z ---------------------------------------------------------------------- 2022-03-17T20:00:18.7738565Z Ran 1 test in 5.410s 2022-03-17T20:00:18.7738742Z 2022-03-17T20:00:18.7738848Z OK 2022-03-17T20:00:18.7739001Z 2022-03-17T20:00:18.7739127Z Generating XML reports... 2022-03-17T20:00:18.7785427Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200013.xml 2022-03-17T20:00:19.8399415Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:19.8414310Z 2022-03-17T20:00:19.8414794Z Running tests... 2022-03-17T20:00:19.8415320Z ---------------------------------------------------------------------- 2022-03-17T20:00:21.3782237Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:21.4216366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94582 2022-03-17T20:00:21.4339471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94583 2022-03-17T20:00:22.2388193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:22.2487313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:24.9076673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:00:24.9077361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:00:25.2426040Z ok (5.401s) 2022-03-17T20:00:25.2426403Z 2022-03-17T20:00:25.2427212Z ---------------------------------------------------------------------- 2022-03-17T20:00:25.2427697Z Ran 1 test in 5.401s 2022-03-17T20:00:25.2427863Z 2022-03-17T20:00:25.2427953Z OK 2022-03-17T20:00:25.2428088Z 2022-03-17T20:00:25.2428202Z Generating XML reports... 2022-03-17T20:00:25.2471243Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200019.xml 2022-03-17T20:00:26.2506749Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:26.2521087Z 2022-03-17T20:00:26.2521461Z Running tests... 2022-03-17T20:00:26.2522370Z ---------------------------------------------------------------------- 2022-03-17T20:00:27.7959202Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:27.8386766Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94709 2022-03-17T20:00:27.8509092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94710 2022-03-17T20:00:28.6833876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:28.7142097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:31.3221643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:00:31.3222240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:00:31.6595994Z ok (5.407s) 2022-03-17T20:00:31.6596195Z 2022-03-17T20:00:31.6596613Z ---------------------------------------------------------------------- 2022-03-17T20:00:31.6596925Z Ran 1 test in 5.407s 2022-03-17T20:00:31.6597106Z 2022-03-17T20:00:31.6597191Z OK 2022-03-17T20:00:31.6597326Z 2022-03-17T20:00:31.6597451Z Generating XML reports... 2022-03-17T20:00:31.6640251Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200026.xml 2022-03-17T20:00:32.7271423Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:32.7286165Z 2022-03-17T20:00:32.7286549Z Running tests... 2022-03-17T20:00:32.7287026Z ---------------------------------------------------------------------- 2022-03-17T20:00:34.2565817Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:34.2994062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94836 2022-03-17T20:00:34.3117044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94837 2022-03-17T20:00:35.1414839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:35.1730420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:37.7878800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:00:37.7879382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:00:38.1202600Z ok (5.391s) 2022-03-17T20:00:38.1202792Z 2022-03-17T20:00:38.1203178Z ---------------------------------------------------------------------- 2022-03-17T20:00:38.1203552Z Ran 1 test in 5.392s 2022-03-17T20:00:38.1203700Z 2022-03-17T20:00:38.1203792Z OK 2022-03-17T20:00:38.1203926Z 2022-03-17T20:00:38.1204058Z Generating XML reports... 2022-03-17T20:00:38.1247012Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200032.xml 2022-03-17T20:00:39.1760729Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:39.1775453Z 2022-03-17T20:00:39.1775712Z Running tests... 2022-03-17T20:00:39.1776123Z ---------------------------------------------------------------------- 2022-03-17T20:00:40.6786319Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:40.7210844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94963 2022-03-17T20:00:40.7331634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94964 2022-03-17T20:00:41.5938337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:41.5940585Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:41.5978784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:41.5982495Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:44.2427827Z ok (5.065s) 2022-03-17T20:00:44.2428052Z 2022-03-17T20:00:44.2428761Z ---------------------------------------------------------------------- 2022-03-17T20:00:44.2429087Z Ran 1 test in 5.065s 2022-03-17T20:00:44.2429253Z 2022-03-17T20:00:44.2429347Z OK 2022-03-17T20:00:44.2429487Z 2022-03-17T20:00:44.2429620Z Generating XML reports... 2022-03-17T20:00:44.2473728Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200039.xml 2022-03-17T20:00:45.3212626Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:45.3227990Z 2022-03-17T20:00:45.3228274Z Running tests... 2022-03-17T20:00:45.3228704Z ---------------------------------------------------------------------- 2022-03-17T20:00:46.8676018Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:46.9106304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95090 2022-03-17T20:00:46.9228553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95091 2022-03-17T20:00:47.7443058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:47.7444233Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:47.7444991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:47.7449052Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:00:50.4310284Z ok (5.108s) 2022-03-17T20:00:50.4310550Z 2022-03-17T20:00:50.4310969Z ---------------------------------------------------------------------- 2022-03-17T20:00:50.4311293Z Ran 1 test in 5.108s 2022-03-17T20:00:50.4311459Z 2022-03-17T20:00:50.4311551Z OK 2022-03-17T20:00:50.4311688Z 2022-03-17T20:00:50.4311806Z Generating XML reports... 2022-03-17T20:00:50.4355891Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200045.xml 2022-03-17T20:00:51.4928022Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:51.4943264Z 2022-03-17T20:00:51.4943492Z Running tests... 2022-03-17T20:00:51.4943939Z ---------------------------------------------------------------------- 2022-03-17T20:00:53.0251363Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:53.0680575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95217 2022-03-17T20:00:53.0806604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95218 2022-03-17T20:00:53.9092561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:00:53.9394643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:00:56.5889660Z ok (5.094s) 2022-03-17T20:00:56.5889934Z 2022-03-17T20:00:56.5890352Z ---------------------------------------------------------------------- 2022-03-17T20:00:56.5890704Z Ran 1 test in 5.095s 2022-03-17T20:00:56.5890873Z 2022-03-17T20:00:56.5890980Z OK 2022-03-17T20:00:56.5891023Z 2022-03-17T20:00:56.5891157Z Generating XML reports... 2022-03-17T20:00:56.5937034Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200051.xml 2022-03-17T20:00:57.6520705Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:00:57.6535901Z 2022-03-17T20:00:57.6536376Z Running tests... 2022-03-17T20:00:57.6536870Z ---------------------------------------------------------------------- 2022-03-17T20:00:59.1930671Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:00:59.2361663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95344 2022-03-17T20:00:59.2487563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95345 2022-03-17T20:01:00.0740213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:00.0997868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:02.7568322Z ok (5.103s) 2022-03-17T20:01:02.7568572Z 2022-03-17T20:01:02.7568979Z ---------------------------------------------------------------------- 2022-03-17T20:01:02.7569318Z Ran 1 test in 5.103s 2022-03-17T20:01:02.7569495Z 2022-03-17T20:01:02.7569608Z OK 2022-03-17T20:01:02.7569719Z 2022-03-17T20:01:02.7569872Z Generating XML reports... 2022-03-17T20:01:02.7612261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200057.xml 2022-03-17T20:01:03.7938752Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:03.7955935Z 2022-03-17T20:01:03.7956394Z Running tests... 2022-03-17T20:01:03.7956913Z ---------------------------------------------------------------------- 2022-03-17T20:01:03.7966155Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest) 2022-03-17T20:01:05.3408382Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:05.3847591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95471 2022-03-17T20:01:05.3972832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95472 2022-03-17T20:01:06.2478758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:06.2592858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:08.2041212Z ok (4.408s) 2022-03-17T20:01:08.2041539Z 2022-03-17T20:01:08.2041931Z ---------------------------------------------------------------------- 2022-03-17T20:01:08.2042263Z Ran 1 test in 4.409s 2022-03-17T20:01:08.2042375Z 2022-03-17T20:01:08.2042451Z OK 2022-03-17T20:01:08.2042569Z 2022-03-17T20:01:08.2042679Z Generating XML reports... 2022-03-17T20:01:08.2085993Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200103.xml 2022-03-17T20:01:09.2713357Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:09.2728467Z 2022-03-17T20:01:09.2728952Z Running tests... 2022-03-17T20:01:09.2729433Z ---------------------------------------------------------------------- 2022-03-17T20:01:09.2738696Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T20:01:10.8166894Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:10.8607801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95597 2022-03-17T20:01:10.8740066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95598 2022-03-17T20:01:11.6941902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:11.6942415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:13.7811447Z ok (4.508s) 2022-03-17T20:01:13.7811765Z 2022-03-17T20:01:13.7812187Z ---------------------------------------------------------------------- 2022-03-17T20:01:13.7812441Z Ran 1 test in 4.508s 2022-03-17T20:01:13.7812581Z 2022-03-17T20:01:13.7812671Z OK 2022-03-17T20:01:13.7813136Z 2022-03-17T20:01:13.7813291Z Generating XML reports... 2022-03-17T20:01:13.7855404Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200109.xml 2022-03-17T20:01:14.8510106Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:14.8525657Z 2022-03-17T20:01:14.8525947Z Running tests... 2022-03-17T20:01:14.8526378Z ---------------------------------------------------------------------- 2022-03-17T20:01:14.8537949Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T20:01:16.4186754Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:16.4626120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95723 2022-03-17T20:01:16.4756870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95724 2022-03-17T20:01:17.2976200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:17.2976713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:19.0188949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.0189510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.0471510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.0471981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.0615477Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:01:19.0616309Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:01:19.0617445Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:01:19.0618276Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:01:19.0722780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.0723213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.0925675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.0926150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.1204167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.1204633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.1447942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.1448357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:19.4828668Z ok (4.630s) 2022-03-17T20:01:19.4828853Z 2022-03-17T20:01:19.4829607Z ---------------------------------------------------------------------- 2022-03-17T20:01:19.4830011Z Ran 1 test in 4.630s 2022-03-17T20:01:19.4830184Z 2022-03-17T20:01:19.4830276Z OK 2022-03-17T20:01:19.4830395Z 2022-03-17T20:01:19.4830524Z Generating XML reports... 2022-03-17T20:01:19.4873030Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200114.xml 2022-03-17T20:01:20.5517224Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:20.5532525Z 2022-03-17T20:01:20.5532778Z Running tests... 2022-03-17T20:01:20.5533440Z ---------------------------------------------------------------------- 2022-03-17T20:01:20.5545053Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T20:01:22.1258299Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:22.1696552Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95849 2022-03-17T20:01:22.1822188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95850 2022-03-17T20:01:22.9842396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:22.9898897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:24.6746669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.6747248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.7047497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.7047943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.7200176Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:01:24.7201041Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:01:24.7202188Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:01:24.7202994Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:01:24.7308786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.7309286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.7516596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.7517077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.7814192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.7814658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.8067794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:24.8068261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:25.0891999Z ok (4.536s) 2022-03-17T20:01:25.0892324Z 2022-03-17T20:01:25.0892948Z ---------------------------------------------------------------------- 2022-03-17T20:01:25.0893272Z Ran 1 test in 4.536s 2022-03-17T20:01:25.0893444Z 2022-03-17T20:01:25.0893546Z OK 2022-03-17T20:01:25.0893680Z 2022-03-17T20:01:25.0893851Z Generating XML reports... 2022-03-17T20:01:25.0937228Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200120.xml 2022-03-17T20:01:26.1048929Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:26.1064819Z 2022-03-17T20:01:26.1065185Z Running tests... 2022-03-17T20:01:26.1065677Z ---------------------------------------------------------------------- 2022-03-17T20:01:26.1074864Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T20:01:27.6318795Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:27.6742283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95975 2022-03-17T20:01:27.6865146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95976 2022-03-17T20:01:28.5242629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:28.5253845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:30.2116428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:30.2117420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:30.2405954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:30.2406880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:30.5935562Z ok (4.487s) 2022-03-17T20:01:30.5935786Z 2022-03-17T20:01:30.5936172Z ---------------------------------------------------------------------- 2022-03-17T20:01:30.5936578Z Ran 1 test in 4.487s 2022-03-17T20:01:30.5936707Z 2022-03-17T20:01:30.5936812Z OK 2022-03-17T20:01:30.5936926Z 2022-03-17T20:01:30.5937083Z Generating XML reports... 2022-03-17T20:01:30.5979790Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200126.xml 2022-03-17T20:01:31.6589218Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:31.6603858Z 2022-03-17T20:01:31.6604134Z Running tests... 2022-03-17T20:01:31.6604826Z ---------------------------------------------------------------------- 2022-03-17T20:01:31.6613357Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T20:01:33.2092130Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:33.2525817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96101 2022-03-17T20:01:33.2650238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96102 2022-03-17T20:01:34.1115307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:34.1335287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:35.8115849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:35.8116454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:35.8431640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:35.8432119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:36.1721703Z ok (4.511s) 2022-03-17T20:01:36.1721844Z 2022-03-17T20:01:36.1722252Z ---------------------------------------------------------------------- 2022-03-17T20:01:36.1722626Z Ran 1 test in 4.512s 2022-03-17T20:01:36.1722767Z 2022-03-17T20:01:36.1722869Z OK 2022-03-17T20:01:36.1723002Z 2022-03-17T20:01:36.1723148Z Generating XML reports... 2022-03-17T20:01:36.1767252Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200131.xml 2022-03-17T20:01:37.2362139Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:37.2376776Z 2022-03-17T20:01:37.2377225Z Running tests... 2022-03-17T20:01:37.2377701Z ---------------------------------------------------------------------- 2022-03-17T20:01:37.2390192Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T20:01:38.7658367Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:38.8086961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96227 2022-03-17T20:01:38.8211443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96228 2022-03-17T20:01:39.6461617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:39.6802700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:41.3163245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:41.3163858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:41.3396538Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:01:41.3398075Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:01:41.3738691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:41.3739257Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:41.7284305Z ok (4.490s) 2022-03-17T20:01:41.7284677Z 2022-03-17T20:01:41.7285206Z ---------------------------------------------------------------------- 2022-03-17T20:01:41.7285539Z Ran 1 test in 4.491s 2022-03-17T20:01:41.7285707Z 2022-03-17T20:01:41.7285800Z OK 2022-03-17T20:01:41.7286880Z 2022-03-17T20:01:41.7287021Z Generating XML reports... 2022-03-17T20:01:41.7330353Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200137.xml 2022-03-17T20:01:42.7924292Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:42.7938894Z 2022-03-17T20:01:42.7939225Z Running tests... 2022-03-17T20:01:42.7939889Z ---------------------------------------------------------------------- 2022-03-17T20:01:42.7953093Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T20:01:44.3383086Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:44.3811723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96353 2022-03-17T20:01:44.3935948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96354 2022-03-17T20:01:45.2357365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:45.2817656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:47.3008611Z ok (4.507s) 2022-03-17T20:01:47.3009043Z 2022-03-17T20:01:47.3010269Z ---------------------------------------------------------------------- 2022-03-17T20:01:47.3010630Z Ran 1 test in 4.507s 2022-03-17T20:01:47.3010776Z 2022-03-17T20:01:47.3010868Z OK 2022-03-17T20:01:47.3011005Z 2022-03-17T20:01:47.3011142Z Generating XML reports... 2022-03-17T20:01:47.3054886Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200142.xml 2022-03-17T20:01:48.3674896Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:48.3689612Z 2022-03-17T20:01:48.3689841Z Running tests... 2022-03-17T20:01:48.3690567Z ---------------------------------------------------------------------- 2022-03-17T20:01:48.3699889Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest) 2022-03-17T20:01:49.9058479Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:49.9488315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96479 2022-03-17T20:01:49.9612331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96480 2022-03-17T20:01:50.8065343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:50.8074085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:52.4711070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:52.4711861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:52.5000064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:52.5000571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:52.7681580Z ok (4.399s) 2022-03-17T20:01:52.7681917Z 2022-03-17T20:01:52.7682489Z ---------------------------------------------------------------------- 2022-03-17T20:01:52.7682853Z Ran 1 test in 4.399s 2022-03-17T20:01:52.7683016Z 2022-03-17T20:01:52.7683107Z OK 2022-03-17T20:01:52.7683240Z 2022-03-17T20:01:52.7683472Z Generating XML reports... 2022-03-17T20:01:52.7725737Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200148.xml 2022-03-17T20:01:53.8313211Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:53.8331025Z 2022-03-17T20:01:53.8331535Z Running tests... 2022-03-17T20:01:53.8332328Z ---------------------------------------------------------------------- 2022-03-17T20:01:53.8345113Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T20:01:55.3687308Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:01:55.4112419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96605 2022-03-17T20:01:55.4241149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96606 2022-03-17T20:01:56.2277142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:01:56.2662631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:01:57.9045963Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:01:57.9077510Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:01:57.9330126Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:01:57.9331742Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:01:57.9333902Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:01:57.9334942Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:01:57.9433921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:57.9434842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:57.9927367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:57.9928282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:01:58.3313040Z ok (4.498s) 2022-03-17T20:01:58.3313279Z 2022-03-17T20:01:58.3313671Z ---------------------------------------------------------------------- 2022-03-17T20:01:58.3314009Z Ran 1 test in 4.498s 2022-03-17T20:01:58.3314241Z 2022-03-17T20:01:58.3314415Z OK 2022-03-17T20:01:58.3314616Z 2022-03-17T20:01:58.3314897Z Generating XML reports... 2022-03-17T20:01:58.3357719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200153.xml 2022-03-17T20:01:59.3437268Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:01:59.3453380Z 2022-03-17T20:01:59.3453698Z Running tests... 2022-03-17T20:01:59.3454430Z ---------------------------------------------------------------------- 2022-03-17T20:01:59.3468459Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T20:02:00.8841583Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:00.9282789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96731 2022-03-17T20:02:00.9405906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96732 2022-03-17T20:02:01.7904204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:01.8029874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:03.4863583Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:02:03.4865220Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:02:03.4866858Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:02:03.4868002Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:02:03.4983113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:03.4986260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:03.5376313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:03.5378739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:03.8479678Z ok (4.502s) 2022-03-17T20:02:03.8479874Z 2022-03-17T20:02:03.8480291Z ---------------------------------------------------------------------- 2022-03-17T20:02:03.8480614Z Ran 1 test in 4.503s 2022-03-17T20:02:03.8480786Z 2022-03-17T20:02:03.8480885Z OK 2022-03-17T20:02:03.8481022Z 2022-03-17T20:02:03.8481158Z Generating XML reports... 2022-03-17T20:02:03.8525381Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200159.xml 2022-03-17T20:02:04.9108966Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:04.9123720Z 2022-03-17T20:02:04.9123967Z Running tests... 2022-03-17T20:02:04.9124432Z ---------------------------------------------------------------------- 2022-03-17T20:02:04.9139090Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest) 2022-03-17T20:02:06.4428343Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:06.4870625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96857 2022-03-17T20:02:06.4995311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96858 2022-03-17T20:02:07.3389344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:07.3828398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:09.0115460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:09.0116637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:09.0445253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:09.0448009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:09.4077222Z ok (4.495s) 2022-03-17T20:02:09.4077421Z 2022-03-17T20:02:09.4077833Z ---------------------------------------------------------------------- 2022-03-17T20:02:09.4078175Z Ran 1 test in 4.495s 2022-03-17T20:02:09.4078341Z 2022-03-17T20:02:09.4078433Z OK 2022-03-17T20:02:09.4078568Z 2022-03-17T20:02:09.4078707Z Generating XML reports... 2022-03-17T20:02:09.4121942Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200204.xml 2022-03-17T20:02:10.4883738Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:10.4899157Z 2022-03-17T20:02:10.4899437Z Running tests... 2022-03-17T20:02:10.4899937Z ---------------------------------------------------------------------- 2022-03-17T20:02:10.4915149Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest) 2022-03-17T20:02:12.0422708Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:12.0853172Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96983 2022-03-17T20:02:12.0982409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96984 2022-03-17T20:02:12.8965869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:12.9177608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:14.5692138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:14.5704714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:14.5985512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:14.5986297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:14.6177558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:14.6178384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:14.6453665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:14.6454156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:14.9052225Z ok (4.415s) 2022-03-17T20:02:14.9052423Z 2022-03-17T20:02:14.9052811Z ---------------------------------------------------------------------- 2022-03-17T20:02:14.9053153Z Ran 1 test in 4.415s 2022-03-17T20:02:14.9053319Z 2022-03-17T20:02:14.9053393Z OK 2022-03-17T20:02:14.9053528Z 2022-03-17T20:02:14.9053663Z Generating XML reports... 2022-03-17T20:02:14.9097455Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200210.xml 2022-03-17T20:02:15.9711555Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:15.9726584Z 2022-03-17T20:02:15.9726984Z Running tests... 2022-03-17T20:02:15.9727527Z ---------------------------------------------------------------------- 2022-03-17T20:02:17.5024434Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:17.5453359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97109 2022-03-17T20:02:17.5575922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97110 2022-03-17T20:02:18.3668060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:18.3879475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:21.0656737Z ok (5.093s) 2022-03-17T20:02:21.0657010Z 2022-03-17T20:02:21.0657467Z ---------------------------------------------------------------------- 2022-03-17T20:02:21.0657800Z Ran 1 test in 5.093s 2022-03-17T20:02:21.0657899Z 2022-03-17T20:02:21.0657948Z OK 2022-03-17T20:02:21.0658067Z 2022-03-17T20:02:21.0658219Z Generating XML reports... 2022-03-17T20:02:21.0700427Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200215.xml 2022-03-17T20:02:22.1293681Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:22.1309193Z 2022-03-17T20:02:22.1309426Z Running tests... 2022-03-17T20:02:22.1309874Z ---------------------------------------------------------------------- 2022-03-17T20:02:23.6870605Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:23.7299133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97236 2022-03-17T20:02:23.7426498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97237 2022-03-17T20:02:24.5677224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:24.5739738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:27.2507114Z ok (5.119s) 2022-03-17T20:02:27.2507538Z 2022-03-17T20:02:27.2508176Z ---------------------------------------------------------------------- 2022-03-17T20:02:27.2508568Z Ran 1 test in 5.120s 2022-03-17T20:02:27.2508677Z 2022-03-17T20:02:27.2508767Z OK 2022-03-17T20:02:27.2508901Z 2022-03-17T20:02:27.2509016Z Generating XML reports... 2022-03-17T20:02:27.2552136Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200222.xml 2022-03-17T20:02:28.3115045Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:28.3130534Z 2022-03-17T20:02:28.3130732Z Running tests... 2022-03-17T20:02:28.3131157Z ---------------------------------------------------------------------- 2022-03-17T20:02:29.8421418Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:29.8850127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97363 2022-03-17T20:02:29.8975022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97364 2022-03-17T20:02:30.7470659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:30.7491131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:33.4058480Z ok (5.092s) 2022-03-17T20:02:33.4058686Z 2022-03-17T20:02:33.4059172Z ---------------------------------------------------------------------- 2022-03-17T20:02:33.4059440Z Ran 1 test in 5.093s 2022-03-17T20:02:33.4059595Z 2022-03-17T20:02:33.4059702Z OK 2022-03-17T20:02:33.4059827Z 2022-03-17T20:02:33.4059961Z Generating XML reports... 2022-03-17T20:02:33.4107125Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200228.xml 2022-03-17T20:02:34.4730116Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:34.4746500Z 2022-03-17T20:02:34.4746758Z Running tests... 2022-03-17T20:02:34.4747209Z ---------------------------------------------------------------------- 2022-03-17T20:02:34.4762146Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest) 2022-03-17T20:02:36.0356807Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:36.0783050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97490 2022-03-17T20:02:36.0912057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97491 2022-03-17T20:02:36.8866874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:36.8893023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:39.4990219Z ok (5.024s) 2022-03-17T20:02:39.4990505Z 2022-03-17T20:02:39.4990921Z ---------------------------------------------------------------------- 2022-03-17T20:02:39.4991244Z Ran 1 test in 5.024s 2022-03-17T20:02:39.4991410Z 2022-03-17T20:02:39.4991508Z OK 2022-03-17T20:02:39.4991610Z 2022-03-17T20:02:39.4991679Z Generating XML reports... 2022-03-17T20:02:39.5033747Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200234.xml 2022-03-17T20:02:40.5583776Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:40.5599365Z 2022-03-17T20:02:40.5599522Z Running tests... 2022-03-17T20:02:40.5600048Z ---------------------------------------------------------------------- 2022-03-17T20:02:40.5609690Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest) 2022-03-17T20:02:42.1062446Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:42.1488880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97617 2022-03-17T20:02:42.1611927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97618 2022-03-17T20:02:42.9592898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:43.0001162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:45.5691711Z ok (5.009s) 2022-03-17T20:02:45.5691956Z 2022-03-17T20:02:45.5692345Z ---------------------------------------------------------------------- 2022-03-17T20:02:45.5692665Z Ran 1 test in 5.009s 2022-03-17T20:02:45.5693071Z 2022-03-17T20:02:45.5693176Z OK 2022-03-17T20:02:45.5693324Z 2022-03-17T20:02:45.5693453Z Generating XML reports... 2022-03-17T20:02:45.5734812Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200240.xml 2022-03-17T20:02:46.6314577Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:46.6329360Z 2022-03-17T20:02:46.6329598Z Running tests... 2022-03-17T20:02:46.6330038Z ---------------------------------------------------------------------- 2022-03-17T20:02:48.1744456Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:48.2174594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97744 2022-03-17T20:02:48.2299049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97745 2022-03-17T20:02:49.0288079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:49.0813717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:49.2334514Z skip (2.600s) 2022-03-17T20:02:49.2334739Z 2022-03-17T20:02:49.2335099Z ---------------------------------------------------------------------- 2022-03-17T20:02:49.2335496Z Ran 1 test in 2.600s 2022-03-17T20:02:49.2335651Z 2022-03-17T20:02:49.2335762Z OK (skipped=1) 2022-03-17T20:02:49.2335917Z 2022-03-17T20:02:49.2336041Z Generating XML reports... 2022-03-17T20:02:49.2379008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200246.xml 2022-03-17T20:02:50.2879532Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:50.2896282Z 2022-03-17T20:02:50.2896641Z Running tests... 2022-03-17T20:02:50.2897513Z ---------------------------------------------------------------------- 2022-03-17T20:02:51.8297907Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:51.8724058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97853 2022-03-17T20:02:51.8847531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97854 2022-03-17T20:02:52.7486242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:52.7499520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:55.1628827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:55.1645017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:55.2192243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:55.2209483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:55.2743015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:55.2760768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:55.3288641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:55.3308008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:02:55.6931843Z ok (5.403s) 2022-03-17T20:02:55.6932178Z 2022-03-17T20:02:55.6932658Z ---------------------------------------------------------------------- 2022-03-17T20:02:55.6932979Z Ran 1 test in 5.404s 2022-03-17T20:02:55.6933127Z 2022-03-17T20:02:55.6933220Z OK 2022-03-17T20:02:55.6933379Z 2022-03-17T20:02:55.6933521Z Generating XML reports... 2022-03-17T20:02:55.6975998Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200250.xml 2022-03-17T20:02:56.7577765Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:02:56.7592813Z 2022-03-17T20:02:56.7593211Z Running tests... 2022-03-17T20:02:56.7593705Z ---------------------------------------------------------------------- 2022-03-17T20:02:58.2948590Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:02:58.3378969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97980 2022-03-17T20:02:58.3506053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97981 2022-03-17T20:02:59.1712508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:02:59.1717521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:02:59.1718878Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-03-17T20:02:59.1719526Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-03-17T20:02:59.1726409Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-03-17T20:02:59.1727085Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-03-17T20:02:59.3540759Z ok (2.594s) 2022-03-17T20:02:59.3541006Z 2022-03-17T20:02:59.3541704Z ---------------------------------------------------------------------- 2022-03-17T20:02:59.3542066Z Ran 1 test in 2.595s 2022-03-17T20:02:59.3542214Z 2022-03-17T20:02:59.3542305Z OK 2022-03-17T20:02:59.3542374Z 2022-03-17T20:02:59.3542504Z Generating XML reports... 2022-03-17T20:02:59.3584059Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200256.xml 2022-03-17T20:03:00.3986953Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:00.4002040Z 2022-03-17T20:03:00.4002292Z Running tests... 2022-03-17T20:03:00.4002673Z ---------------------------------------------------------------------- 2022-03-17T20:03:01.9153161Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:01.9586753Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98093 2022-03-17T20:03:01.9711991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98094 2022-03-17T20:03:02.8068399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:02.8237674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:05.4793542Z ok (5.079s) 2022-03-17T20:03:05.4794001Z 2022-03-17T20:03:05.4794614Z ---------------------------------------------------------------------- 2022-03-17T20:03:05.4795235Z Ran 1 test in 5.079s 2022-03-17T20:03:05.4795527Z 2022-03-17T20:03:05.4795701Z OK 2022-03-17T20:03:05.4795951Z 2022-03-17T20:03:05.4796178Z Generating XML reports... 2022-03-17T20:03:05.4840549Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200300.xml 2022-03-17T20:03:06.5357199Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:06.5371623Z 2022-03-17T20:03:06.5371894Z Running tests... 2022-03-17T20:03:06.5372309Z ---------------------------------------------------------------------- 2022-03-17T20:03:08.0478205Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:08.0901376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98220 2022-03-17T20:03:08.1025750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98221 2022-03-17T20:03:08.9104021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:08.9649083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:11.6105541Z ok (5.073s) 2022-03-17T20:03:11.6105748Z 2022-03-17T20:03:11.6106154Z ---------------------------------------------------------------------- 2022-03-17T20:03:11.6106496Z Ran 1 test in 5.073s 2022-03-17T20:03:11.6106644Z 2022-03-17T20:03:11.6106743Z OK 2022-03-17T20:03:11.6106879Z 2022-03-17T20:03:11.6107017Z Generating XML reports... 2022-03-17T20:03:11.6150361Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200306.xml 2022-03-17T20:03:12.6696694Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:12.6711929Z 2022-03-17T20:03:12.6712175Z Running tests... 2022-03-17T20:03:12.6712638Z ---------------------------------------------------------------------- 2022-03-17T20:03:14.2105738Z test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:14.2534747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98347 2022-03-17T20:03:14.2658290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98348 2022-03-17T20:03:15.1362298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:15.1363323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:17.7911412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:03:17.7911940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:03:17.8407038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:03:17.8407558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:03:18.1747935Z ok (5.503s) 2022-03-17T20:03:18.1748186Z 2022-03-17T20:03:18.1748584Z ---------------------------------------------------------------------- 2022-03-17T20:03:18.1748931Z Ran 1 test in 5.504s 2022-03-17T20:03:18.1749080Z 2022-03-17T20:03:18.1749168Z OK 2022-03-17T20:03:18.1749264Z 2022-03-17T20:03:18.1749419Z Generating XML reports... 2022-03-17T20:03:18.1791969Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200312.xml 2022-03-17T20:03:19.2410907Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:19.2427083Z 2022-03-17T20:03:19.2427375Z Running tests... 2022-03-17T20:03:19.2427816Z ---------------------------------------------------------------------- 2022-03-17T20:03:20.7963230Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:20.8398452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98485 2022-03-17T20:03:20.8526380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98486 2022-03-17T20:03:21.6566101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:21.7059168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:21.7179800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:21.7180652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:21.7181552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:21.7182457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:23.7597006Z ok (4.517s) 2022-03-17T20:03:23.7597293Z 2022-03-17T20:03:23.7597717Z ---------------------------------------------------------------------- 2022-03-17T20:03:23.7598164Z Ran 1 test in 4.517s 2022-03-17T20:03:23.7598488Z 2022-03-17T20:03:23.7598579Z OK 2022-03-17T20:03:23.7598722Z 2022-03-17T20:03:23.7598859Z Generating XML reports... 2022-03-17T20:03:23.7643445Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200319.xml 2022-03-17T20:03:24.8309284Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:24.8324320Z 2022-03-17T20:03:24.8324465Z Running tests... 2022-03-17T20:03:24.8325278Z ---------------------------------------------------------------------- 2022-03-17T20:03:26.3802220Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:26.4234057Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98617 2022-03-17T20:03:26.4359012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98618 2022-03-17T20:03:27.2339179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:27.2350431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:27.2479864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:27.2491789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:27.2492990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:27.2554844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:29.2425946Z ok (4.410s) 2022-03-17T20:03:29.2426271Z 2022-03-17T20:03:29.2426683Z ---------------------------------------------------------------------- 2022-03-17T20:03:29.2427027Z Ran 1 test in 4.410s 2022-03-17T20:03:29.2427173Z 2022-03-17T20:03:29.2428570Z OK 2022-03-17T20:03:29.2429006Z 2022-03-17T20:03:29.2429299Z Generating XML reports... 2022-03-17T20:03:29.2470924Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200324.xml 2022-03-17T20:03:30.3058390Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:30.3074434Z 2022-03-17T20:03:30.3074577Z Running tests... 2022-03-17T20:03:30.3075523Z ---------------------------------------------------------------------- 2022-03-17T20:03:31.8524608Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:31.8955671Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98743 2022-03-17T20:03:31.9077992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98744 2022-03-17T20:03:32.7798845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:32.7809432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:32.7841486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:32.7854297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:32.7855250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:32.7912961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:34.7145609Z ok (4.407s) 2022-03-17T20:03:34.7145837Z 2022-03-17T20:03:34.7146233Z ---------------------------------------------------------------------- 2022-03-17T20:03:34.7146570Z Ran 1 test in 4.407s 2022-03-17T20:03:34.7146734Z 2022-03-17T20:03:34.7146825Z OK 2022-03-17T20:03:34.7146949Z 2022-03-17T20:03:34.7147084Z Generating XML reports... 2022-03-17T20:03:34.7189128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200330.xml 2022-03-17T20:03:35.7696910Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:35.7712305Z 2022-03-17T20:03:35.7712771Z Running tests... 2022-03-17T20:03:35.7713291Z ---------------------------------------------------------------------- 2022-03-17T20:03:37.2962237Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:37.3389849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98869 2022-03-17T20:03:37.3511330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98870 2022-03-17T20:03:38.2199882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:38.2308757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:38.2519371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:38.2519800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:38.2520587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:38.2521282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:40.2581410Z ok (4.487s) 2022-03-17T20:03:40.2581632Z 2022-03-17T20:03:40.2582062Z ---------------------------------------------------------------------- 2022-03-17T20:03:40.2582369Z Ran 1 test in 4.487s 2022-03-17T20:03:40.2582536Z 2022-03-17T20:03:40.2583955Z OK 2022-03-17T20:03:40.2584843Z 2022-03-17T20:03:40.2584982Z Generating XML reports... 2022-03-17T20:03:40.2625249Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200335.xml 2022-03-17T20:03:41.3249338Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:41.3265121Z 2022-03-17T20:03:41.3265426Z Running tests... 2022-03-17T20:03:41.3265870Z ---------------------------------------------------------------------- 2022-03-17T20:03:42.8666781Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:42.9095168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99001 2022-03-17T20:03:42.9220760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99002 2022-03-17T20:03:43.7309137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:43.7321659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:43.7821223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:43.7836092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:43.7836872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:43.7930857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:45.7287400Z ok (4.402s) 2022-03-17T20:03:45.7287653Z 2022-03-17T20:03:45.7288296Z ---------------------------------------------------------------------- 2022-03-17T20:03:45.7288643Z Ran 1 test in 4.402s 2022-03-17T20:03:45.7288809Z 2022-03-17T20:03:45.7288908Z OK 2022-03-17T20:03:45.7289024Z 2022-03-17T20:03:45.7289159Z Generating XML reports... 2022-03-17T20:03:45.7338421Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200341.xml 2022-03-17T20:03:46.7913386Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:46.7928870Z 2022-03-17T20:03:46.7929375Z Running tests... 2022-03-17T20:03:46.7929804Z ---------------------------------------------------------------------- 2022-03-17T20:03:48.3257882Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:48.3687231Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99127 2022-03-17T20:03:48.3810708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99128 2022-03-17T20:03:49.1850567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:49.1863593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:03:49.2048878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:49.2062321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:03:49.2063444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:49.2067521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:03:51.1880281Z ok (4.395s) 2022-03-17T20:03:51.1880639Z 2022-03-17T20:03:51.1881310Z ---------------------------------------------------------------------- 2022-03-17T20:03:51.1881902Z Ran 1 test in 4.395s 2022-03-17T20:03:51.1882190Z 2022-03-17T20:03:51.1882351Z OK 2022-03-17T20:03:51.1882635Z 2022-03-17T20:03:51.1882846Z Generating XML reports... 2022-03-17T20:03:51.1925964Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200346.xml 2022-03-17T20:03:52.2440374Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:52.2457321Z 2022-03-17T20:03:52.2457710Z Running tests... 2022-03-17T20:03:52.2458185Z ---------------------------------------------------------------------- 2022-03-17T20:03:53.8018248Z test_fp16 (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:03:53.8444240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99253 2022-03-17T20:03:53.8578805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99254 2022-03-17T20:03:54.7052496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:03:54.7307886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:03:57.6665678Z ok (5.421s) 2022-03-17T20:03:57.6666268Z 2022-03-17T20:03:57.6666673Z ---------------------------------------------------------------------- 2022-03-17T20:03:57.6666952Z Ran 1 test in 5.421s 2022-03-17T20:03:57.6667118Z 2022-03-17T20:03:57.6667210Z OK 2022-03-17T20:03:57.6667346Z 2022-03-17T20:03:57.6667463Z Generating XML reports... 2022-03-17T20:03:57.6709086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200352.xml 2022-03-17T20:03:58.6871353Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:03:58.6886059Z 2022-03-17T20:03:58.6886647Z Running tests... 2022-03-17T20:03:58.6887531Z ---------------------------------------------------------------------- 2022-03-17T20:04:00.2266305Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:00.2698472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99380 2022-03-17T20:04:00.2825477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99381 2022-03-17T20:04:01.0841586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:01.0843109Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:01.1255956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:01.1258718Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:03.7907295Z ok (5.102s) 2022-03-17T20:04:03.7907641Z 2022-03-17T20:04:03.7908359Z ---------------------------------------------------------------------- 2022-03-17T20:04:03.7908842Z Ran 1 test in 5.102s 2022-03-17T20:04:03.7909079Z 2022-03-17T20:04:03.7909177Z OK 2022-03-17T20:04:03.7909245Z 2022-03-17T20:04:03.7909382Z Generating XML reports... 2022-03-17T20:04:03.7951698Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200358.xml 2022-03-17T20:04:04.8576871Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:04.8592791Z 2022-03-17T20:04:04.8593324Z Running tests... 2022-03-17T20:04:04.8593740Z ---------------------------------------------------------------------- 2022-03-17T20:04:06.3953976Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:06.4387124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99507 2022-03-17T20:04:06.4513277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99508 2022-03-17T20:04:07.2815199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:07.2816600Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:07.3075554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:07.3077792Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:09.9595199Z ok (5.100s) 2022-03-17T20:04:09.9595486Z 2022-03-17T20:04:09.9595839Z ---------------------------------------------------------------------- 2022-03-17T20:04:09.9596162Z Ran 1 test in 5.100s 2022-03-17T20:04:09.9596278Z 2022-03-17T20:04:09.9596371Z OK 2022-03-17T20:04:09.9596508Z 2022-03-17T20:04:09.9596624Z Generating XML reports... 2022-03-17T20:04:09.9638765Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200404.xml 2022-03-17T20:04:11.0206084Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:11.0220790Z 2022-03-17T20:04:11.0221496Z Running tests... 2022-03-17T20:04:11.0221975Z ---------------------------------------------------------------------- 2022-03-17T20:04:12.5717341Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:12.6150280Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99634 2022-03-17T20:04:12.6276592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99635 2022-03-17T20:04:13.4763229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:13.4811709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:16.4364386Z ok (5.414s) 2022-03-17T20:04:16.4366192Z 2022-03-17T20:04:16.4367129Z ---------------------------------------------------------------------- 2022-03-17T20:04:16.4367413Z Ran 1 test in 5.414s 2022-03-17T20:04:16.4367572Z 2022-03-17T20:04:16.4368139Z OK 2022-03-17T20:04:16.4368421Z 2022-03-17T20:04:16.4368580Z Generating XML reports... 2022-03-17T20:04:16.4414779Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200411.xml 2022-03-17T20:04:17.5074557Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:17.5089623Z 2022-03-17T20:04:17.5090062Z Running tests... 2022-03-17T20:04:17.5090541Z ---------------------------------------------------------------------- 2022-03-17T20:04:19.0645410Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:19.1077746Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99761 2022-03-17T20:04:19.1206088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99762 2022-03-17T20:04:19.9610290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:20.0016257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:23.0293322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.0309729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.0584149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.0584648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.0856551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.0857021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.1139919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.1141527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.1414581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.1415092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.1693943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.1694952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.1972231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.1977557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.2263980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.2269927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.2555073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.2560792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.2859502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.2865667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.3159229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.3164914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.3461563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.3467382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.3749889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.3755585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.4036912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.4042223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.4321661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.4326616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.4618731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.4625262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.4908219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.4914255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.5202611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:23.5209471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:24.0311646Z ok (6.522s) 2022-03-17T20:04:24.0311871Z 2022-03-17T20:04:24.0312306Z ---------------------------------------------------------------------- 2022-03-17T20:04:24.0312889Z Ran 1 test in 6.522s 2022-03-17T20:04:24.0313145Z 2022-03-17T20:04:24.0313300Z OK 2022-03-17T20:04:24.0313505Z 2022-03-17T20:04:24.0313712Z Generating XML reports... 2022-03-17T20:04:24.0357318Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200417.xml 2022-03-17T20:04:25.0939613Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:25.0954706Z 2022-03-17T20:04:25.0955068Z Running tests... 2022-03-17T20:04:25.0955992Z ---------------------------------------------------------------------- 2022-03-17T20:04:26.6351609Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:26.6782182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99888 2022-03-17T20:04:26.6906187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99889 2022-03-17T20:04:27.5323029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:27.5418565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:27.6941884Z skip (2.598s) 2022-03-17T20:04:27.6942106Z 2022-03-17T20:04:27.6942518Z ---------------------------------------------------------------------- 2022-03-17T20:04:27.6942849Z Ran 1 test in 2.599s 2022-03-17T20:04:27.6943006Z 2022-03-17T20:04:27.6944356Z OK (skipped=1) 2022-03-17T20:04:27.6945278Z 2022-03-17T20:04:27.6945433Z Generating XML reports... 2022-03-17T20:04:27.6985857Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200425.xml 2022-03-17T20:04:28.7347531Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:28.7362898Z 2022-03-17T20:04:28.7363404Z Running tests... 2022-03-17T20:04:28.7363895Z ---------------------------------------------------------------------- 2022-03-17T20:04:30.2568133Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:30.2994954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99997 2022-03-17T20:04:30.3118823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99998 2022-03-17T20:04:31.1153706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:31.1158888Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1159908Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1160879Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1161864Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1162846Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1163828Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1698266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:31.1703860Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1704844Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1706120Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1707185Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1708137Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.1709115Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:04:31.3153857Z ok (2.579s) 2022-03-17T20:04:31.3154076Z 2022-03-17T20:04:31.3154471Z ---------------------------------------------------------------------- 2022-03-17T20:04:31.3154817Z Ran 1 test in 2.579s 2022-03-17T20:04:31.3154983Z 2022-03-17T20:04:31.3155085Z OK 2022-03-17T20:04:31.3155203Z 2022-03-17T20:04:31.3155338Z Generating XML reports... 2022-03-17T20:04:31.3198925Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200428.xml 2022-03-17T20:04:32.3487146Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:32.3501738Z 2022-03-17T20:04:32.3501984Z Running tests... 2022-03-17T20:04:32.3502857Z ---------------------------------------------------------------------- 2022-03-17T20:04:33.8590343Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:33.9014214Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100106 2022-03-17T20:04:33.9141467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100107 2022-03-17T20:04:34.7224869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:34.7559441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:37.6222432Z ok (5.272s) 2022-03-17T20:04:37.6222649Z 2022-03-17T20:04:37.6223084Z ---------------------------------------------------------------------- 2022-03-17T20:04:37.6223393Z Ran 1 test in 5.272s 2022-03-17T20:04:37.6223546Z 2022-03-17T20:04:37.6223639Z OK 2022-03-17T20:04:37.6223768Z 2022-03-17T20:04:37.6223883Z Generating XML reports... 2022-03-17T20:04:37.6265369Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200432.xml 2022-03-17T20:04:38.6782745Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:38.6798576Z 2022-03-17T20:04:38.6798897Z Running tests... 2022-03-17T20:04:38.6799373Z ---------------------------------------------------------------------- 2022-03-17T20:04:40.2172923Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:40.2604975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100233 2022-03-17T20:04:40.2729915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100234 2022-03-17T20:04:41.0952776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:41.1148641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:44.0816278Z ok (5.401s) 2022-03-17T20:04:44.0816494Z 2022-03-17T20:04:44.0816914Z ---------------------------------------------------------------------- 2022-03-17T20:04:44.0817238Z Ran 1 test in 5.402s 2022-03-17T20:04:44.0817527Z 2022-03-17T20:04:44.0817618Z OK 2022-03-17T20:04:44.0817754Z 2022-03-17T20:04:44.0817892Z Generating XML reports... 2022-03-17T20:04:44.0860754Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200438.xml 2022-03-17T20:04:45.1393241Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:45.1407875Z 2022-03-17T20:04:45.1408156Z Running tests... 2022-03-17T20:04:45.1408603Z ---------------------------------------------------------------------- 2022-03-17T20:04:46.6756055Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:46.7194492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100360 2022-03-17T20:04:46.7320770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100361 2022-03-17T20:04:47.5842905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:47.6116354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:50.2090110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:50.2090585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:50.5407345Z ok (5.400s) 2022-03-17T20:04:50.5407599Z 2022-03-17T20:04:50.5408004Z ---------------------------------------------------------------------- 2022-03-17T20:04:50.5408334Z Ran 1 test in 5.400s 2022-03-17T20:04:50.5408494Z 2022-03-17T20:04:50.5408513Z OK 2022-03-17T20:04:50.5408647Z 2022-03-17T20:04:50.5408775Z Generating XML reports... 2022-03-17T20:04:50.5451017Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200445.xml 2022-03-17T20:04:51.6022846Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:51.6038517Z 2022-03-17T20:04:51.6038845Z Running tests... 2022-03-17T20:04:51.6039309Z ---------------------------------------------------------------------- 2022-03-17T20:04:53.1554112Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:53.1983481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100487 2022-03-17T20:04:53.2114305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100488 2022-03-17T20:04:54.0474376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:04:54.0536567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:04:56.6828408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:56.6828994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:04:57.0201939Z ok (5.416s) 2022-03-17T20:04:57.0202146Z 2022-03-17T20:04:57.0202515Z ---------------------------------------------------------------------- 2022-03-17T20:04:57.0202944Z Ran 1 test in 5.416s 2022-03-17T20:04:57.0203119Z 2022-03-17T20:04:57.0203191Z OK 2022-03-17T20:04:57.0203334Z 2022-03-17T20:04:57.0203422Z Generating XML reports... 2022-03-17T20:04:57.0247631Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200451.xml 2022-03-17T20:04:58.0862603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:04:58.0878531Z 2022-03-17T20:04:58.0878914Z Running tests... 2022-03-17T20:04:58.0879445Z ---------------------------------------------------------------------- 2022-03-17T20:04:59.6239122Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:04:59.6671468Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100614 2022-03-17T20:04:59.6799820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100615 2022-03-17T20:05:00.5128094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:00.5416111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:00.6835760Z skip (2.595s) 2022-03-17T20:05:00.6836119Z 2022-03-17T20:05:00.6836901Z ---------------------------------------------------------------------- 2022-03-17T20:05:00.6837437Z Ran 1 test in 2.596s 2022-03-17T20:05:00.6837518Z 2022-03-17T20:05:00.6837693Z OK (skipped=1) 2022-03-17T20:05:00.6838105Z 2022-03-17T20:05:00.6838388Z Generating XML reports... 2022-03-17T20:05:00.6880396Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200458.xml 2022-03-17T20:05:01.7373107Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:05:01.7388666Z 2022-03-17T20:05:01.7388997Z Running tests... 2022-03-17T20:05:01.7389442Z ---------------------------------------------------------------------- 2022-03-17T20:05:03.2865752Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:05:03.3298606Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100723 2022-03-17T20:05:03.3424278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100724 2022-03-17T20:05:04.1663561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:04.1939867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:04.3460127Z skip (2.607s) 2022-03-17T20:05:04.3460626Z 2022-03-17T20:05:04.3461018Z ---------------------------------------------------------------------- 2022-03-17T20:05:04.3461361Z Ran 1 test in 2.607s 2022-03-17T20:05:04.3461525Z 2022-03-17T20:05:04.3461619Z OK (skipped=1) 2022-03-17T20:05:04.3461800Z 2022-03-17T20:05:04.3461930Z Generating XML reports... 2022-03-17T20:05:04.3505558Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200501.xml 2022-03-17T20:05:05.3905057Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:05:05.3920070Z 2022-03-17T20:05:05.3920521Z Running tests... 2022-03-17T20:05:05.3921024Z ---------------------------------------------------------------------- 2022-03-17T20:05:06.9434243Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:05:06.9873486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100832 2022-03-17T20:05:07.0001456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100833 2022-03-17T20:05:07.8076846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:07.8105600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:09.4062795Z ok (4.014s) 2022-03-17T20:05:09.4063601Z 2022-03-17T20:05:09.4063997Z ---------------------------------------------------------------------- 2022-03-17T20:05:09.4064666Z Ran 1 test in 4.014s 2022-03-17T20:05:09.4064821Z 2022-03-17T20:05:09.4064923Z OK 2022-03-17T20:05:09.4065046Z 2022-03-17T20:05:09.4065170Z Generating XML reports... 2022-03-17T20:05:09.4108314Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200505.xml 2022-03-17T20:05:10.4631733Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:05:10.4646413Z 2022-03-17T20:05:10.4646666Z Running tests... 2022-03-17T20:05:10.4647111Z ---------------------------------------------------------------------- 2022-03-17T20:05:11.9990612Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:05:12.0415947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100947 2022-03-17T20:05:12.0541141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100948 2022-03-17T20:05:12.8634557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:12.8961381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:13.0575646Z skip (2.593s) 2022-03-17T20:05:13.0575861Z 2022-03-17T20:05:13.0576266Z ---------------------------------------------------------------------- 2022-03-17T20:05:13.0576592Z Ran 1 test in 2.593s 2022-03-17T20:05:13.0576755Z 2022-03-17T20:05:13.0576818Z OK (skipped=1) 2022-03-17T20:05:13.0576976Z 2022-03-17T20:05:13.0577082Z Generating XML reports... 2022-03-17T20:05:13.0619290Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200510.xml 2022-03-17T20:05:14.0644044Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:05:14.0659915Z 2022-03-17T20:05:14.0660089Z Running tests... 2022-03-17T20:05:14.0660859Z ---------------------------------------------------------------------- 2022-03-17T20:05:15.6213907Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:05:15.6647023Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101056 2022-03-17T20:05:15.6773279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101057 2022-03-17T20:05:16.5085699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:16.5129439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:19.1587076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:05:19.1587624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:05:19.4857498Z ok (5.419s) 2022-03-17T20:05:19.4857996Z 2022-03-17T20:05:19.4858781Z ---------------------------------------------------------------------- 2022-03-17T20:05:19.4859389Z Ran 1 test in 5.420s 2022-03-17T20:05:19.4859668Z 2022-03-17T20:05:19.4859858Z OK 2022-03-17T20:05:19.4860112Z 2022-03-17T20:05:19.4860697Z Generating XML reports... 2022-03-17T20:05:19.4904887Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200514.xml 2022-03-17T20:05:20.5511300Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:05:20.5526143Z 2022-03-17T20:05:20.5526682Z Running tests... 2022-03-17T20:05:20.5527177Z ---------------------------------------------------------------------- 2022-03-17T20:05:22.0986369Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:05:22.1416267Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101183 2022-03-17T20:05:22.1542477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101184 2022-03-17T20:05:22.9603450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:22.9610373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:25.6049518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:05:25.6050022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:05:25.9630839Z ok (5.410s) 2022-03-17T20:05:25.9631087Z 2022-03-17T20:05:25.9631523Z ---------------------------------------------------------------------- 2022-03-17T20:05:25.9632093Z Ran 1 test in 5.410s 2022-03-17T20:05:25.9632281Z 2022-03-17T20:05:25.9632377Z OK 2022-03-17T20:05:25.9632548Z 2022-03-17T20:05:25.9632655Z Generating XML reports... 2022-03-17T20:05:25.9675800Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200520.xml 2022-03-17T20:05:27.0299500Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:05:27.0315008Z 2022-03-17T20:05:27.0315258Z Running tests... 2022-03-17T20:05:27.0315704Z ---------------------------------------------------------------------- 2022-03-17T20:05:28.5563972Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:05:28.5995395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101310 2022-03-17T20:05:28.6118428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101311 2022-03-17T20:05:29.4244866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:29.4534703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:48.2483538Z ok (21.216s) 2022-03-17T20:05:48.2483783Z 2022-03-17T20:05:48.2484170Z ---------------------------------------------------------------------- 2022-03-17T20:05:48.2486259Z Ran 1 test in 21.217s 2022-03-17T20:05:48.2486461Z 2022-03-17T20:05:48.2486566Z OK 2022-03-17T20:05:48.2486744Z 2022-03-17T20:05:48.2486848Z Generating XML reports... 2022-03-17T20:05:48.2539374Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200527.xml 2022-03-17T20:05:49.2968120Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:05:49.2984075Z 2022-03-17T20:05:49.2984223Z Running tests... 2022-03-17T20:05:49.2984918Z ---------------------------------------------------------------------- 2022-03-17T20:05:49.3007087Z test_no_grad (__main__.DistributedDataParallelTest) 2022-03-17T20:05:50.8299390Z Note: this test can be sped up by only running it on a CPU module ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:05:50.8726945Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101437 2022-03-17T20:05:50.8852990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101438 2022-03-17T20:05:51.7294836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:05:51.7552280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:54.6953775Z ok (5.397s) 2022-03-17T20:05:54.6953993Z 2022-03-17T20:05:54.6954334Z ---------------------------------------------------------------------- 2022-03-17T20:05:54.6954662Z Ran 1 test in 5.397s 2022-03-17T20:05:54.6954816Z 2022-03-17T20:05:54.6954911Z OK 2022-03-17T20:05:54.6955051Z 2022-03-17T20:05:54.6955205Z Generating XML reports... 2022-03-17T20:05:54.6997689Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200549.xml 2022-03-17T20:05:55.7565659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:05:55.7581025Z 2022-03-17T20:05:55.7581388Z Running tests... 2022-03-17T20:05:55.7581805Z ---------------------------------------------------------------------- 2022-03-17T20:05:57.3120746Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:05:57.3554333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101560 2022-03-17T20:05:57.3684675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101561 2022-03-17T20:05:58.2080137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:05:58.2336341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:00.8771263Z ok (5.119s) 2022-03-17T20:06:00.8771488Z 2022-03-17T20:06:00.8771904Z ---------------------------------------------------------------------- 2022-03-17T20:06:00.8772220Z Ran 1 test in 5.119s 2022-03-17T20:06:00.8772402Z 2022-03-17T20:06:00.8772493Z OK 2022-03-17T20:06:00.8772628Z 2022-03-17T20:06:00.8772762Z Generating XML reports... 2022-03-17T20:06:00.8815503Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200555.xml 2022-03-17T20:06:01.9431752Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:01.9446134Z 2022-03-17T20:06:01.9446439Z Running tests... 2022-03-17T20:06:01.9447343Z ---------------------------------------------------------------------- 2022-03-17T20:06:03.4711296Z test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:03.5139525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101683 2022-03-17T20:06:03.5264525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101684 2022-03-17T20:06:04.3562684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:04.3566694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:04.3841759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:04.3847515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:04.3848346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:04.3871948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:04.5299962Z ok (2.585s) 2022-03-17T20:06:04.5300181Z 2022-03-17T20:06:04.5300798Z ---------------------------------------------------------------------- 2022-03-17T20:06:04.5301119Z Ran 1 test in 2.585s 2022-03-17T20:06:04.5301274Z 2022-03-17T20:06:04.5301383Z OK 2022-03-17T20:06:04.5301466Z 2022-03-17T20:06:04.5301600Z Generating XML reports... 2022-03-17T20:06:04.5343012Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200601.xml 2022-03-17T20:06:05.5739314Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:05.5755223Z 2022-03-17T20:06:05.5755808Z Running tests... 2022-03-17T20:06:05.5756323Z ---------------------------------------------------------------------- 2022-03-17T20:06:07.1231645Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:07.1661719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101796 2022-03-17T20:06:07.1786164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101797 2022-03-17T20:06:08.0030407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:08.0032179Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:08.0115741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:08.0118621Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:10.3655381Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:10.3656444Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:10.3703196Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:10.3704177Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:10.3750940Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:10.3751909Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:10.6867501Z ok (5.111s) 2022-03-17T20:06:10.6867820Z 2022-03-17T20:06:10.6868369Z ---------------------------------------------------------------------- 2022-03-17T20:06:10.6868610Z Ran 1 test in 5.111s 2022-03-17T20:06:10.6868778Z 2022-03-17T20:06:10.6868850Z OK 2022-03-17T20:06:10.6868998Z 2022-03-17T20:06:10.6869212Z Generating XML reports... 2022-03-17T20:06:10.6912520Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200605.xml 2022-03-17T20:06:11.7570195Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:11.7587031Z 2022-03-17T20:06:11.7587398Z Running tests... 2022-03-17T20:06:11.7587811Z ---------------------------------------------------------------------- 2022-03-17T20:06:13.2873487Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:13.3304943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101923 2022-03-17T20:06:13.3430696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101924 2022-03-17T20:06:14.1703016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:14.1705262Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:14.2042925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:14.2045919Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:16.4879253Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:16.4880347Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:16.4926841Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:16.4927755Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:16.4973981Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:16.4974885Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:06:16.8514231Z ok (5.092s) 2022-03-17T20:06:16.8514452Z 2022-03-17T20:06:16.8514871Z ---------------------------------------------------------------------- 2022-03-17T20:06:16.8515195Z Ran 1 test in 5.093s 2022-03-17T20:06:16.8515366Z 2022-03-17T20:06:16.8515461Z OK 2022-03-17T20:06:16.8515610Z 2022-03-17T20:06:16.8515726Z Generating XML reports... 2022-03-17T20:06:16.8558986Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200611.xml 2022-03-17T20:06:17.9221425Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:17.9236802Z 2022-03-17T20:06:17.9236935Z Running tests... 2022-03-17T20:06:17.9237391Z ---------------------------------------------------------------------- 2022-03-17T20:06:19.4662659Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:19.5098328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102050 2022-03-17T20:06:19.5224420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102051 2022-03-17T20:06:19.5355115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102052 2022-03-17T20:06:20.3367930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:20.3914630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:20.3951434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:06:20.5395926Z skip (2.616s) 2022-03-17T20:06:20.5396133Z 2022-03-17T20:06:20.5396511Z ---------------------------------------------------------------------- 2022-03-17T20:06:20.5396845Z Ran 1 test in 2.616s 2022-03-17T20:06:20.5397010Z 2022-03-17T20:06:20.5397101Z OK (skipped=1) 2022-03-17T20:06:20.5397255Z 2022-03-17T20:06:20.5397380Z Generating XML reports... 2022-03-17T20:06:20.5440394Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200617.xml 2022-03-17T20:06:21.5641575Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:21.5656247Z 2022-03-17T20:06:21.5656384Z Running tests... 2022-03-17T20:06:21.5656868Z ---------------------------------------------------------------------- 2022-03-17T20:06:23.0981962Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:23.1412244Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102195 2022-03-17T20:06:23.1535714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102196 2022-03-17T20:06:23.1663782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102197 2022-03-17T20:06:23.9665182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:23.9757842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:06:23.9914082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:24.1702714Z skip (2.604s) 2022-03-17T20:06:24.1703154Z 2022-03-17T20:06:24.1703619Z ---------------------------------------------------------------------- 2022-03-17T20:06:24.1703981Z Ran 1 test in 2.605s 2022-03-17T20:06:24.1704151Z 2022-03-17T20:06:24.1704256Z OK (skipped=1) 2022-03-17T20:06:24.1704486Z 2022-03-17T20:06:24.1704603Z Generating XML reports... 2022-03-17T20:06:24.1747211Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200621.xml 2022-03-17T20:06:25.2188982Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:25.2203926Z 2022-03-17T20:06:25.2204158Z Running tests... 2022-03-17T20:06:25.2205051Z ---------------------------------------------------------------------- 2022-03-17T20:06:25.2210870Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip (0.001s) 2022-03-17T20:06:25.2211148Z 2022-03-17T20:06:25.2211658Z ---------------------------------------------------------------------- 2022-03-17T20:06:25.2212014Z Ran 1 test in 0.001s 2022-03-17T20:06:25.2212139Z 2022-03-17T20:06:25.2212186Z OK (skipped=1) 2022-03-17T20:06:25.2212341Z 2022-03-17T20:06:25.2212471Z Generating XML reports... 2022-03-17T20:06:25.2247527Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200625.xml 2022-03-17T20:06:26.1322980Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:26.1337899Z 2022-03-17T20:06:26.1338286Z Running tests... 2022-03-17T20:06:26.1338771Z ---------------------------------------------------------------------- 2022-03-17T20:06:27.6708552Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:27.7136775Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102375 2022-03-17T20:06:27.7261388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102376 2022-03-17T20:06:27.7389563Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102377 2022-03-17T20:06:28.5378042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:28.5782181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:28.5918301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:06:28.7428845Z skip (2.609s) 2022-03-17T20:06:28.7429071Z 2022-03-17T20:06:28.7429505Z ---------------------------------------------------------------------- 2022-03-17T20:06:28.7429821Z Ran 1 test in 2.609s 2022-03-17T20:06:28.7429987Z 2022-03-17T20:06:28.7430107Z OK (skipped=1) 2022-03-17T20:06:28.7430493Z 2022-03-17T20:06:28.7430620Z Generating XML reports... 2022-03-17T20:06:28.7473580Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200626.xml 2022-03-17T20:06:29.7975952Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:29.7991104Z 2022-03-17T20:06:29.7991588Z Running tests... 2022-03-17T20:06:29.7992084Z ---------------------------------------------------------------------- 2022-03-17T20:06:31.3341293Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:31.3789369Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102520 2022-03-17T20:06:31.3915837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102521 2022-03-17T20:06:31.4041595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102522 2022-03-17T20:06:32.2126547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:32.2477785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:32.2969045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:06:32.4081874Z ok (2.609s) 2022-03-17T20:06:32.4082157Z 2022-03-17T20:06:32.4082723Z ---------------------------------------------------------------------- 2022-03-17T20:06:32.4083067Z Ran 1 test in 2.609s 2022-03-17T20:06:32.4083207Z 2022-03-17T20:06:32.4083283Z OK 2022-03-17T20:06:32.4083490Z 2022-03-17T20:06:32.4083637Z Generating XML reports... 2022-03-17T20:06:32.4124772Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200629.xml 2022-03-17T20:06:33.4624683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:33.4639826Z 2022-03-17T20:06:33.4640151Z Running tests... 2022-03-17T20:06:33.4640605Z ---------------------------------------------------------------------- 2022-03-17T20:06:35.0050469Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:35.0487700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102665 2022-03-17T20:06:35.0616204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102666 2022-03-17T20:06:35.0747920Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102667 2022-03-17T20:06:35.8746383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:35.9078391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:35.9256306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:06:36.0788958Z ok (2.615s) 2022-03-17T20:06:36.0789192Z 2022-03-17T20:06:36.0789610Z ---------------------------------------------------------------------- 2022-03-17T20:06:36.0789931Z Ran 1 test in 2.615s 2022-03-17T20:06:36.0790099Z 2022-03-17T20:06:36.0790188Z OK 2022-03-17T20:06:36.0790322Z 2022-03-17T20:06:36.0790462Z Generating XML reports... 2022-03-17T20:06:36.0832107Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200633.xml 2022-03-17T20:06:37.1417406Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:37.1433085Z 2022-03-17T20:06:37.1433321Z Running tests... 2022-03-17T20:06:37.1433770Z ---------------------------------------------------------------------- 2022-03-17T20:06:38.6930174Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:38.7373734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102810 2022-03-17T20:06:38.7506868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102811 2022-03-17T20:06:38.7637445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102812 2022-03-17T20:06:39.5761840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:39.5762429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:39.5992849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:06:39.7678955Z ok (2.624s) 2022-03-17T20:06:39.7679164Z 2022-03-17T20:06:39.7679541Z ---------------------------------------------------------------------- 2022-03-17T20:06:39.7679879Z Ran 1 test in 2.624s 2022-03-17T20:06:39.7680044Z 2022-03-17T20:06:39.7680115Z OK 2022-03-17T20:06:39.7680229Z 2022-03-17T20:06:39.7680336Z Generating XML reports... 2022-03-17T20:06:39.7722222Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200637.xml 2022-03-17T20:06:40.8263452Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:40.8278462Z 2022-03-17T20:06:40.8278876Z Running tests... 2022-03-17T20:06:40.8279419Z ---------------------------------------------------------------------- 2022-03-17T20:06:40.8294968Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip (0.001s) 2022-03-17T20:06:40.8295217Z 2022-03-17T20:06:40.8295571Z ---------------------------------------------------------------------- 2022-03-17T20:06:40.8295889Z Ran 1 test in 0.002s 2022-03-17T20:06:40.8296030Z 2022-03-17T20:06:40.8296109Z OK (skipped=1) 2022-03-17T20:06:40.8296265Z 2022-03-17T20:06:40.8296399Z Generating XML reports... 2022-03-17T20:06:40.8331085Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200640.xml 2022-03-17T20:06:41.7462235Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:41.7477599Z 2022-03-17T20:06:41.7477850Z Running tests... 2022-03-17T20:06:41.7478309Z ---------------------------------------------------------------------- 2022-03-17T20:06:43.3177329Z test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:43.3617938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102990 2022-03-17T20:06:43.3744545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102991 2022-03-17T20:06:43.3874114Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102992 2022-03-17T20:06:44.1933282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:44.2812041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2 2022-03-17T20:06:44.2812532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:44.4915148Z skip (2.743s) 2022-03-17T20:06:44.4915357Z 2022-03-17T20:06:44.4915765Z ---------------------------------------------------------------------- 2022-03-17T20:06:44.4916097Z Ran 1 test in 2.744s 2022-03-17T20:06:44.4916266Z 2022-03-17T20:06:44.4916374Z OK (skipped=1) 2022-03-17T20:06:44.4916806Z 2022-03-17T20:06:44.4916939Z Generating XML reports... 2022-03-17T20:06:44.4959915Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200641.xml 2022-03-17T20:06:45.5407848Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:45.5423206Z 2022-03-17T20:06:45.5423480Z Running tests... 2022-03-17T20:06:45.5423928Z ---------------------------------------------------------------------- 2022-03-17T20:06:45.5430886Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip (0.001s) 2022-03-17T20:06:45.5431148Z 2022-03-17T20:06:45.5431765Z ---------------------------------------------------------------------- 2022-03-17T20:06:45.5432058Z Ran 1 test in 0.001s 2022-03-17T20:06:45.5432236Z 2022-03-17T20:06:45.5432341Z OK (skipped=1) 2022-03-17T20:06:45.5432498Z 2022-03-17T20:06:45.5432625Z Generating XML reports... 2022-03-17T20:06:45.5467900Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220317200645.xml 2022-03-17T20:06:46.4565779Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:46.4580882Z 2022-03-17T20:06:46.4581043Z Running tests... 2022-03-17T20:06:46.4581759Z ---------------------------------------------------------------------- 2022-03-17T20:06:48.0004547Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:48.0439127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103170 2022-03-17T20:06:48.0567438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103171 2022-03-17T20:06:48.8900412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:48.8901906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:48.9329712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:48.9333984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:48.9334797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:48.9410163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:50.5630305Z ok (4.105s) 2022-03-17T20:06:50.5630524Z 2022-03-17T20:06:50.5630924Z ---------------------------------------------------------------------- 2022-03-17T20:06:50.5631264Z Ran 1 test in 4.105s 2022-03-17T20:06:50.5631421Z 2022-03-17T20:06:50.5631512Z OK 2022-03-17T20:06:50.5631643Z 2022-03-17T20:06:50.5631774Z Generating XML reports... 2022-03-17T20:06:50.5674372Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200646.xml 2022-03-17T20:06:51.6339429Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:51.6354861Z 2022-03-17T20:06:51.6355120Z Running tests... 2022-03-17T20:06:51.6355577Z ---------------------------------------------------------------------- 2022-03-17T20:06:53.1808215Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:53.2243597Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103285 2022-03-17T20:06:53.2368453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103286 2022-03-17T20:06:54.0386390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:06:54.0389377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:06:54.0438181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:06:54.0441631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:06:54.0442410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:54.0492511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:06:56.7450929Z ok (5.109s) 2022-03-17T20:06:56.7451320Z 2022-03-17T20:06:56.7451737Z ---------------------------------------------------------------------- 2022-03-17T20:06:56.7452064Z Ran 1 test in 5.110s 2022-03-17T20:06:56.7452219Z 2022-03-17T20:06:56.7452315Z OK 2022-03-17T20:06:56.7452713Z 2022-03-17T20:06:56.7452864Z Generating XML reports... 2022-03-17T20:06:56.7494959Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200651.xml 2022-03-17T20:06:57.7902140Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:06:57.7917909Z 2022-03-17T20:06:57.7918194Z Running tests... 2022-03-17T20:06:57.7918902Z ---------------------------------------------------------------------- 2022-03-17T20:06:59.3373467Z test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:06:59.3802567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103408 2022-03-17T20:06:59.3927918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103409 2022-03-17T20:07:00.2284757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:00.2287745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:00.2422951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:00.2427052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:00.2428116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:00.2491903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:02.9010696Z ok (5.109s) 2022-03-17T20:07:02.9010902Z 2022-03-17T20:07:02.9011563Z ---------------------------------------------------------------------- 2022-03-17T20:07:02.9011904Z Ran 1 test in 5.109s 2022-03-17T20:07:02.9012040Z 2022-03-17T20:07:02.9012115Z OK 2022-03-17T20:07:02.9012248Z 2022-03-17T20:07:02.9012383Z Generating XML reports... 2022-03-17T20:07:02.9054333Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200657.xml 2022-03-17T20:07:03.9643963Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:03.9659284Z 2022-03-17T20:07:03.9659561Z Running tests... 2022-03-17T20:07:03.9660003Z ---------------------------------------------------------------------- 2022-03-17T20:07:05.4913790Z test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:05.5346216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103531 2022-03-17T20:07:05.5471777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103532 2022-03-17T20:07:06.3547290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:06.3551071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:06.3754233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:06.3758062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:06.3758861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:06.3857714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:09.0553009Z ok (5.089s) 2022-03-17T20:07:09.0553206Z 2022-03-17T20:07:09.0553631Z ---------------------------------------------------------------------- 2022-03-17T20:07:09.0553973Z Ran 1 test in 5.089s 2022-03-17T20:07:09.0554123Z 2022-03-17T20:07:09.0554184Z OK 2022-03-17T20:07:09.0554381Z 2022-03-17T20:07:09.0554479Z Generating XML reports... 2022-03-17T20:07:09.0598346Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200703.xml 2022-03-17T20:07:10.0972573Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:10.0988095Z 2022-03-17T20:07:10.0988243Z Running tests... 2022-03-17T20:07:10.0988920Z ---------------------------------------------------------------------- 2022-03-17T20:07:11.6265288Z test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:11.6697427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103654 2022-03-17T20:07:11.6820856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103655 2022-03-17T20:07:12.5171621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:12.5174457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:12.5547109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:12.5551086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:12.5551826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:12.5580320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:15.1916972Z ok (5.093s) 2022-03-17T20:07:15.1917204Z 2022-03-17T20:07:15.1917574Z ---------------------------------------------------------------------- 2022-03-17T20:07:15.1917916Z Ran 1 test in 5.093s 2022-03-17T20:07:15.1918081Z 2022-03-17T20:07:15.1918170Z OK 2022-03-17T20:07:15.1918287Z 2022-03-17T20:07:15.1918422Z Generating XML reports... 2022-03-17T20:07:15.1960894Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200710.xml 2022-03-17T20:07:16.2375377Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:16.2390371Z 2022-03-17T20:07:16.2390796Z Running tests... 2022-03-17T20:07:16.2391261Z ---------------------------------------------------------------------- 2022-03-17T20:07:17.7752698Z test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:17.8181680Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103777 2022-03-17T20:07:17.8307466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103778 2022-03-17T20:07:18.6452853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:18.6453396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:18.6455984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:18.6456760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:18.6457495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:18.6458143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:21.2385516Z ok (4.999s) 2022-03-17T20:07:21.2385927Z 2022-03-17T20:07:21.2386609Z ---------------------------------------------------------------------- 2022-03-17T20:07:21.2386964Z Ran 1 test in 4.999s 2022-03-17T20:07:21.2387129Z 2022-03-17T20:07:21.2387222Z OK 2022-03-17T20:07:21.2387400Z 2022-03-17T20:07:21.2387516Z Generating XML reports... 2022-03-17T20:07:21.2429665Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200716.xml 2022-03-17T20:07:22.2936430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:22.2951795Z 2022-03-17T20:07:22.2951958Z Running tests... 2022-03-17T20:07:22.2952386Z ---------------------------------------------------------------------- 2022-03-17T20:07:23.8375418Z test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:23.8802523Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103900 2022-03-17T20:07:23.8927359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103901 2022-03-17T20:07:24.7232044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:24.7234445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:24.7284724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:24.7288691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:24.7289717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:24.7337571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:27.4009938Z ok (5.105s) 2022-03-17T20:07:27.4010192Z 2022-03-17T20:07:27.4010598Z ---------------------------------------------------------------------- 2022-03-17T20:07:27.4010957Z Ran 1 test in 5.106s 2022-03-17T20:07:27.4011127Z 2022-03-17T20:07:27.4011212Z OK 2022-03-17T20:07:27.4011337Z 2022-03-17T20:07:27.4011471Z Generating XML reports... 2022-03-17T20:07:27.4055548Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200722.xml 2022-03-17T20:07:28.4693704Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:28.4709497Z 2022-03-17T20:07:28.4709877Z Running tests... 2022-03-17T20:07:28.4710314Z ---------------------------------------------------------------------- 2022-03-17T20:07:30.0080882Z test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:30.0510281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104023 2022-03-17T20:07:30.0635169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104024 2022-03-17T20:07:30.9043162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:30.9045520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:30.9077588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:30.9081301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:30.9082241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:30.9148464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:32.4697264Z ok (3.998s) 2022-03-17T20:07:32.4697505Z 2022-03-17T20:07:32.4698501Z ---------------------------------------------------------------------- 2022-03-17T20:07:32.4698802Z Ran 1 test in 3.999s 2022-03-17T20:07:32.4698969Z 2022-03-17T20:07:32.4699061Z OK 2022-03-17T20:07:32.4699206Z 2022-03-17T20:07:32.4699340Z Generating XML reports... 2022-03-17T20:07:32.4742918Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200728.xml 2022-03-17T20:07:33.5363758Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:33.5378458Z 2022-03-17T20:07:33.5378706Z Running tests... 2022-03-17T20:07:33.5379162Z ---------------------------------------------------------------------- 2022-03-17T20:07:35.0584099Z test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:35.1012500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104138 2022-03-17T20:07:35.1136917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104139 2022-03-17T20:07:35.9115917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:35.9118511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:35.9165605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:35.9169513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:35.9170662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:35.9221469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:38.6219219Z ok (5.084s) 2022-03-17T20:07:38.6219431Z 2022-03-17T20:07:38.6219850Z ---------------------------------------------------------------------- 2022-03-17T20:07:38.6220164Z Ran 1 test in 5.084s 2022-03-17T20:07:38.6220511Z 2022-03-17T20:07:38.6220616Z OK 2022-03-17T20:07:38.6220779Z 2022-03-17T20:07:38.6220873Z Generating XML reports... 2022-03-17T20:07:38.6265365Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200733.xml 2022-03-17T20:07:39.6681899Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:39.6696882Z 2022-03-17T20:07:39.6697379Z Running tests... 2022-03-17T20:07:39.6697878Z ---------------------------------------------------------------------- 2022-03-17T20:07:41.2196003Z test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:41.2626743Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104261 2022-03-17T20:07:41.2751536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104262 2022-03-17T20:07:42.0798239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:42.0800771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:42.1159233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:42.1163144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:42.1164469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:42.1207623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:45.3842666Z ok (5.714s) 2022-03-17T20:07:45.3842868Z 2022-03-17T20:07:45.3843294Z ---------------------------------------------------------------------- 2022-03-17T20:07:45.3843612Z Ran 1 test in 5.714s 2022-03-17T20:07:45.3843753Z 2022-03-17T20:07:45.3844168Z OK 2022-03-17T20:07:45.3846495Z 2022-03-17T20:07:45.3846757Z Generating XML reports... 2022-03-17T20:07:45.3888847Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200739.xml 2022-03-17T20:07:46.4494488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:46.4510298Z 2022-03-17T20:07:46.4510637Z Running tests... 2022-03-17T20:07:46.4511309Z ---------------------------------------------------------------------- 2022-03-17T20:07:47.9953119Z test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:48.0383774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104384 2022-03-17T20:07:48.0507045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104385 2022-03-17T20:07:48.8586812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:48.8588879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:48.8873576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:48.8877162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:48.8878106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:48.8894246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:51.5589547Z ok (5.108s) 2022-03-17T20:07:51.5589833Z 2022-03-17T20:07:51.5590232Z ---------------------------------------------------------------------- 2022-03-17T20:07:51.5590527Z Ran 1 test in 5.108s 2022-03-17T20:07:51.5590698Z 2022-03-17T20:07:51.5590801Z OK 2022-03-17T20:07:51.5590903Z 2022-03-17T20:07:51.5591034Z Generating XML reports... 2022-03-17T20:07:51.5634821Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200746.xml 2022-03-17T20:07:52.6110069Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:52.6125280Z 2022-03-17T20:07:52.6125491Z Running tests... 2022-03-17T20:07:52.6125944Z ---------------------------------------------------------------------- 2022-03-17T20:07:54.1387228Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:54.1816558Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104507 2022-03-17T20:07:54.1940731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104508 2022-03-17T20:07:55.0157638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:07:55.0160348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:07:55.0305324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:07:55.0309204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:07:55.0309994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:55.0364594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:07:56.6012867Z ok (3.988s) 2022-03-17T20:07:56.6013236Z 2022-03-17T20:07:56.6013673Z ---------------------------------------------------------------------- 2022-03-17T20:07:56.6013958Z Ran 1 test in 3.989s 2022-03-17T20:07:56.6014125Z 2022-03-17T20:07:56.6014217Z OK 2022-03-17T20:07:56.6014352Z 2022-03-17T20:07:56.6014468Z Generating XML reports... 2022-03-17T20:07:56.6058608Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200752.xml 2022-03-17T20:07:57.6643741Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:07:57.6658988Z 2022-03-17T20:07:57.6659412Z Running tests... 2022-03-17T20:07:57.6659917Z ---------------------------------------------------------------------- 2022-03-17T20:07:59.2259732Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:07:59.2692364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104622 2022-03-17T20:07:59.2820052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104623 2022-03-17T20:08:00.0840548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:00.0842946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:00.1252269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:00.1256310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:00.1257112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:00.1350745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:02.7901457Z ok (5.124s) 2022-03-17T20:08:02.7901853Z 2022-03-17T20:08:02.7902318Z ---------------------------------------------------------------------- 2022-03-17T20:08:02.7902606Z Ran 1 test in 5.124s 2022-03-17T20:08:02.7902796Z 2022-03-17T20:08:02.7902889Z OK 2022-03-17T20:08:02.7903022Z 2022-03-17T20:08:02.7903154Z Generating XML reports... 2022-03-17T20:08:02.7945174Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200757.xml 2022-03-17T20:08:03.8601334Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:08:03.8616684Z 2022-03-17T20:08:03.8616937Z Running tests... 2022-03-17T20:08:03.8617362Z ---------------------------------------------------------------------- 2022-03-17T20:08:05.3969482Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:05.4412574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104745 2022-03-17T20:08:05.4539973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104746 2022-03-17T20:08:06.3402708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:06.3403234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:06.3404588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:06.3406808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:06.3407569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:06.3517877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:08.9620891Z ok (5.100s) 2022-03-17T20:08:08.9621105Z 2022-03-17T20:08:08.9621809Z ---------------------------------------------------------------------- 2022-03-17T20:08:08.9622153Z Ran 1 test in 5.100s 2022-03-17T20:08:08.9622315Z 2022-03-17T20:08:08.9622394Z OK 2022-03-17T20:08:08.9622516Z 2022-03-17T20:08:08.9622661Z Generating XML reports... 2022-03-17T20:08:08.9666011Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200803.xml 2022-03-17T20:08:10.0068543Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:08:10.0083193Z 2022-03-17T20:08:10.0083346Z Running tests... 2022-03-17T20:08:10.0084066Z ---------------------------------------------------------------------- 2022-03-17T20:08:11.5227614Z test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:11.5656122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104868 2022-03-17T20:08:11.5781666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104869 2022-03-17T20:08:12.3835065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:12.3837880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:12.4254073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:12.4257791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:12.4258584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:12.4344817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:13.9841149Z ok (3.975s) 2022-03-17T20:08:13.9841337Z 2022-03-17T20:08:13.9841703Z ---------------------------------------------------------------------- 2022-03-17T20:08:13.9842043Z Ran 1 test in 3.976s 2022-03-17T20:08:13.9842148Z 2022-03-17T20:08:13.9842222Z OK 2022-03-17T20:08:13.9842361Z 2022-03-17T20:08:13.9842494Z Generating XML reports... 2022-03-17T20:08:13.9885866Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200810.xml 2022-03-17T20:08:15.0486657Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:08:15.0502382Z 2022-03-17T20:08:15.0502882Z Running tests... 2022-03-17T20:08:15.0503320Z ---------------------------------------------------------------------- 2022-03-17T20:08:16.5814161Z test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:16.6244811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104983 2022-03-17T20:08:16.6372113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104984 2022-03-17T20:08:17.4787588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:17.4789963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:17.5010173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:17.5013603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:17.5014426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:17.5095385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:20.1453176Z ok (5.095s) 2022-03-17T20:08:20.1453421Z 2022-03-17T20:08:20.1453797Z ---------------------------------------------------------------------- 2022-03-17T20:08:20.1454116Z Ran 1 test in 5.095s 2022-03-17T20:08:20.1454282Z 2022-03-17T20:08:20.1454376Z OK 2022-03-17T20:08:20.1454511Z 2022-03-17T20:08:20.1454628Z Generating XML reports... 2022-03-17T20:08:20.1497299Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200815.xml 2022-03-17T20:08:21.1897289Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:08:21.1912306Z 2022-03-17T20:08:21.1912428Z Running tests... 2022-03-17T20:08:21.1913412Z ---------------------------------------------------------------------- 2022-03-17T20:08:22.7232030Z test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:22.7663414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105106 2022-03-17T20:08:22.7786555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105107 2022-03-17T20:08:23.5307228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:08:23.5310250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:08:23.5824936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:08:23.5829012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:23.5829728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:23.5919080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:08:26.8894637Z ok (5.698s) 2022-03-17T20:08:26.8894857Z 2022-03-17T20:08:26.8895242Z ---------------------------------------------------------------------- 2022-03-17T20:08:26.8895591Z Ran 1 test in 5.698s 2022-03-17T20:08:26.8895759Z 2022-03-17T20:08:26.8895842Z OK 2022-03-17T20:08:26.8895959Z 2022-03-17T20:08:26.8896097Z Generating XML reports... 2022-03-17T20:08:26.8939711Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200821.xml 2022-03-17T20:08:27.9535229Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:08:27.9550130Z 2022-03-17T20:08:27.9550275Z Running tests... 2022-03-17T20:08:27.9551016Z ---------------------------------------------------------------------- 2022-03-17T20:08:29.5084619Z test_common_errors (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:29.5267206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:29.5268414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:08:29.5293234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:29.5294233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:08:29.5317054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:29.5317787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:08:29.5338883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:29.5339994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:08:29.5429067Z ok (1.588s) 2022-03-17T20:08:29.5429378Z 2022-03-17T20:08:29.5429768Z ---------------------------------------------------------------------- 2022-03-17T20:08:29.5430084Z Ran 1 test in 1.588s 2022-03-17T20:08:29.5430267Z 2022-03-17T20:08:29.5430362Z OK 2022-03-17T20:08:29.5430498Z 2022-03-17T20:08:29.5430623Z Generating XML reports... 2022-03-17T20:08:29.5464191Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220317200827.xml 2022-03-17T20:08:30.5056393Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl 2022-03-17T20:08:30.5071436Z 2022-03-17T20:08:30.5071745Z Running tests... 2022-03-17T20:08:30.5072197Z ---------------------------------------------------------------------- 2022-03-17T20:08:32.0630660Z test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:32.0798327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:32.0799234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:08:34.0872932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:08:34.0874176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T20:08:35.1026461Z ok (4.595s) 2022-03-17T20:08:35.1026663Z 2022-03-17T20:08:35.1027213Z ---------------------------------------------------------------------- 2022-03-17T20:08:35.1027602Z Ran 1 test in 4.595s 2022-03-17T20:08:35.1027780Z 2022-03-17T20:08:35.1027887Z OK 2022-03-17T20:08:35.1028005Z 2022-03-17T20:08:35.1028155Z Generating XML reports... 2022-03-17T20:08:35.1070951Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220317200830.xml 2022-03-17T20:08:35.4506928Z Running distributed/test_c10d_spawn_gloo ... [2022-03-17 20:08:35.450081] 2022-03-17T20:08:35.4507900Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:08:35.450191] 2022-03-17T20:08:36.2567526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylvn3que 2022-03-17T20:08:36.2568755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylvn3que/_remote_module_non_sriptable.py 2022-03-17T20:08:37.7898537Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:37.7945841Z , <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]> 2022-03-17T20:08:37.7947239Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) 2022-03-17T20:08:37.7947683Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) 2022-03-17T20:08:37.7948431Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) 2022-03-17T20:08:37.7950075Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]> 2022-03-17T20:08:37.7951375Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:08:37.7951860Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:08:37.7952381Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:08:37.7952947Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:08:37.7953314Z 2022-03-17T20:08:37.7953618Z 2022-03-17T20:08:37.7954763Z , <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]> 2022-03-17T20:08:37.7955930Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:08:37.7956324Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:08:37.7956707Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:08:37.7957393Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:08:37.7957812Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:08:37.7958162Z test_gather (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:08:37.7958525Z test_reduce (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:08:37.7958944Z test_scatter (__main__.TestDistributedNNFunctionsGloo) 2022-03-17T20:08:38.6036906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppzmcqg89 2022-03-17T20:08:38.6037879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppzmcqg89/_remote_module_non_sriptable.py 2022-03-17T20:08:40.1411030Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:40.1478105Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:08:40.1493763Z 2022-03-17T20:08:40.1494124Z Running tests... 2022-03-17T20:08:40.1494543Z ---------------------------------------------------------------------- 2022-03-17T20:08:40.1567337Z test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:40.1722103Z ok (0.023s) 2022-03-17T20:08:40.1723632Z 2022-03-17T20:08:40.1723950Z ---------------------------------------------------------------------- 2022-03-17T20:08:40.1724301Z Ran 1 test in 0.023s 2022-03-17T20:08:40.1724457Z 2022-03-17T20:08:40.1724552Z OK 2022-03-17T20:08:40.1724685Z 2022-03-17T20:08:40.1724819Z Generating XML reports... 2022-03-17T20:08:40.1756961Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200840.xml 2022-03-17T20:08:41.1792407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qs_69q1 2022-03-17T20:08:41.1793189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qs_69q1/_remote_module_non_sriptable.py 2022-03-17T20:08:42.7272345Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:42.7339664Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:08:42.7355815Z 2022-03-17T20:08:42.7356254Z Running tests... 2022-03-17T20:08:42.7356734Z ---------------------------------------------------------------------- 2022-03-17T20:08:43.0309337Z test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:43.0512890Z ok (0.316s) 2022-03-17T20:08:43.0514360Z 2022-03-17T20:08:43.0514935Z ---------------------------------------------------------------------- 2022-03-17T20:08:43.0515309Z Ran 1 test in 0.316s 2022-03-17T20:08:43.0515456Z 2022-03-17T20:08:43.0515558Z OK 2022-03-17T20:08:43.0515693Z 2022-03-17T20:08:43.0515853Z Generating XML reports... 2022-03-17T20:08:43.0550625Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200842.xml 2022-03-17T20:08:44.0773733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17mhxk76 2022-03-17T20:08:44.0776044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17mhxk76/_remote_module_non_sriptable.py 2022-03-17T20:08:45.6261437Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:45.6328878Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:08:45.6344068Z 2022-03-17T20:08:45.6344629Z Running tests... 2022-03-17T20:08:45.6345185Z ---------------------------------------------------------------------- 2022-03-17T20:08:46.4166270Z test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:08:46.8481221Z ok (1.213s) 2022-03-17T20:08:46.8481736Z 2022-03-17T20:08:46.8482294Z ---------------------------------------------------------------------- 2022-03-17T20:08:46.8482821Z Ran 1 test in 1.214s 2022-03-17T20:08:46.8482976Z 2022-03-17T20:08:46.8483070Z OK 2022-03-17T20:08:46.8483206Z 2022-03-17T20:08:46.8483339Z Generating XML reports... 2022-03-17T20:08:46.8518154Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200845.xml 2022-03-17T20:08:48.0028826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk07g3kq6 2022-03-17T20:08:48.0029936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk07g3kq6/_remote_module_non_sriptable.py 2022-03-17T20:08:49.5352438Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:49.5418072Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:08:49.5433885Z 2022-03-17T20:08:49.5434230Z Running tests... 2022-03-17T20:08:49.5434705Z ---------------------------------------------------------------------- 2022-03-17T20:08:50.3839140Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegjvd0h4 2022-03-17T20:08:50.3840321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegjvd0h4/_remote_module_non_sriptable.py 2022-03-17T20:08:50.4184981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v7jvhcs 2022-03-17T20:08:50.4186156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v7jvhcs/_remote_module_non_sriptable.py 2022-03-17T20:08:51.9632041Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:51.9781544Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:52.2966175Z ok (2.753s) 2022-03-17T20:08:52.2966838Z 2022-03-17T20:08:52.2967289Z ---------------------------------------------------------------------- 2022-03-17T20:08:52.2967698Z Ran 1 test in 2.753s 2022-03-17T20:08:52.2967864Z 2022-03-17T20:08:52.2967934Z OK 2022-03-17T20:08:52.2968076Z 2022-03-17T20:08:52.2968164Z Generating XML reports... 2022-03-17T20:08:52.3022643Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200849.xml 2022-03-17T20:08:53.3342024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkna8b2x8 2022-03-17T20:08:53.3343325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkna8b2x8/_remote_module_non_sriptable.py 2022-03-17T20:08:54.8735125Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:54.8802709Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:08:54.8817943Z 2022-03-17T20:08:54.8818197Z Running tests... 2022-03-17T20:08:54.8818632Z ---------------------------------------------------------------------- 2022-03-17T20:08:56.7461634Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1k20ld2 2022-03-17T20:08:56.7463223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1k20ld2/_remote_module_non_sriptable.py 2022-03-17T20:08:56.7896524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8au2qc4 2022-03-17T20:08:56.7897656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8au2qc4/_remote_module_non_sriptable.py 2022-03-17T20:08:58.3369516Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:58.3408017Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:08:59.8343849Z ok (4.952s) 2022-03-17T20:08:59.8345052Z 2022-03-17T20:08:59.8345659Z ---------------------------------------------------------------------- 2022-03-17T20:08:59.8346059Z Ran 1 test in 4.953s 2022-03-17T20:08:59.8346293Z 2022-03-17T20:08:59.8346467Z OK 2022-03-17T20:08:59.8346686Z 2022-03-17T20:08:59.8346904Z Generating XML reports... 2022-03-17T20:08:59.8401267Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200854.xml 2022-03-17T20:09:00.9257920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j91uw13 2022-03-17T20:09:00.9259028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j91uw13/_remote_module_non_sriptable.py 2022-03-17T20:09:02.4503379Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:02.4571739Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:02.4587303Z 2022-03-17T20:09:02.4587733Z Running tests... 2022-03-17T20:09:02.4588225Z ---------------------------------------------------------------------- 2022-03-17T20:09:04.2941349Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_akctd9 2022-03-17T20:09:04.2942662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_akctd9/_remote_module_non_sriptable.py 2022-03-17T20:09:04.3379832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusxj4pm8 2022-03-17T20:09:04.3381400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusxj4pm8/_remote_module_non_sriptable.py 2022-03-17T20:09:05.8875543Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:05.8888509Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:07.3874438Z ok (4.928s) 2022-03-17T20:09:07.3874765Z 2022-03-17T20:09:07.3875315Z ---------------------------------------------------------------------- 2022-03-17T20:09:07.3875928Z Ran 1 test in 4.929s 2022-03-17T20:09:07.3876085Z 2022-03-17T20:09:07.3876179Z OK 2022-03-17T20:09:07.3876317Z 2022-03-17T20:09:07.3876454Z Generating XML reports... 2022-03-17T20:09:07.3929724Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200902.xml 2022-03-17T20:09:08.4858524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0f5g292t 2022-03-17T20:09:08.4859151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0f5g292t/_remote_module_non_sriptable.py 2022-03-17T20:09:10.0221529Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:10.0288592Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:10.0304654Z 2022-03-17T20:09:10.0305097Z Running tests... 2022-03-17T20:09:10.0305790Z ---------------------------------------------------------------------- 2022-03-17T20:09:11.8618683Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_d5c1iue 2022-03-17T20:09:11.8619888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_d5c1iue/_remote_module_non_sriptable.py 2022-03-17T20:09:11.8642394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw7_ardv 2022-03-17T20:09:11.8645458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw7_ardv/_remote_module_non_sriptable.py 2022-03-17T20:09:13.4594124Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:13.4620924Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:14.9410804Z ok (4.910s) 2022-03-17T20:09:14.9411735Z 2022-03-17T20:09:14.9412134Z ---------------------------------------------------------------------- 2022-03-17T20:09:14.9412488Z Ran 1 test in 4.911s 2022-03-17T20:09:14.9412635Z 2022-03-17T20:09:14.9412730Z OK 2022-03-17T20:09:14.9412869Z 2022-03-17T20:09:14.9413007Z Generating XML reports... 2022-03-17T20:09:14.9466973Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200910.xml 2022-03-17T20:09:16.0397457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yj7be41 2022-03-17T20:09:16.0398255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yj7be41/_remote_module_non_sriptable.py 2022-03-17T20:09:17.5794984Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:17.5860201Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:17.5876092Z 2022-03-17T20:09:17.5876377Z Running tests... 2022-03-17T20:09:17.5877055Z ---------------------------------------------------------------------- 2022-03-17T20:09:17.6237765Z test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105973 2022-03-17T20:09:17.6342336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105974 2022-03-17T20:09:18.4512691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmup7132 2022-03-17T20:09:18.4513339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bc9rziz 2022-03-17T20:09:18.4514065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmup7132/_remote_module_non_sriptable.py 2022-03-17T20:09:18.4514568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bc9rziz/_remote_module_non_sriptable.py 2022-03-17T20:09:20.0498158Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:20.0544835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:20.0657571Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:20.0702741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:20.0914328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:20.0914861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:20.0915657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:20.0916369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:21.4426183Z ok (3.855s) 2022-03-17T20:09:21.4426402Z 2022-03-17T20:09:21.4426796Z ---------------------------------------------------------------------- 2022-03-17T20:09:21.4427133Z Ran 1 test in 3.855s 2022-03-17T20:09:21.4427298Z 2022-03-17T20:09:21.4427389Z OK 2022-03-17T20:09:21.4427524Z 2022-03-17T20:09:21.4427639Z Generating XML reports... 2022-03-17T20:09:21.4483352Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200917.xml 2022-03-17T20:09:22.4773790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6pc3wvd 2022-03-17T20:09:22.4774686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6pc3wvd/_remote_module_non_sriptable.py 2022-03-17T20:09:24.0075953Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:24.0142516Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:24.0158270Z 2022-03-17T20:09:24.0158539Z Running tests... 2022-03-17T20:09:24.0158982Z ---------------------------------------------------------------------- 2022-03-17T20:09:24.0521535Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106095 2022-03-17T20:09:24.0626196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106096 2022-03-17T20:09:24.8980643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3_bz992 2022-03-17T20:09:24.8981965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3_bz992/_remote_module_non_sriptable.py 2022-03-17T20:09:24.8984801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08nk0sfg 2022-03-17T20:09:24.8987426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08nk0sfg/_remote_module_non_sriptable.py 2022-03-17T20:09:26.5046387Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:26.5091859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:26.5127911Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:26.5172853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:26.5303898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:26.5304919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:26.5305904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:26.5306597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:27.8712854Z ok (3.855s) 2022-03-17T20:09:27.8713638Z 2022-03-17T20:09:27.8714320Z ---------------------------------------------------------------------- 2022-03-17T20:09:27.8714684Z Ran 1 test in 3.855s 2022-03-17T20:09:27.8714859Z 2022-03-17T20:09:27.8714949Z OK 2022-03-17T20:09:27.8715070Z 2022-03-17T20:09:27.8715212Z Generating XML reports... 2022-03-17T20:09:27.8770839Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200924.xml 2022-03-17T20:09:28.9229347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hqfvcem 2022-03-17T20:09:28.9230406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hqfvcem/_remote_module_non_sriptable.py 2022-03-17T20:09:30.4461274Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:30.4529212Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:30.4545888Z 2022-03-17T20:09:30.4546382Z Running tests... 2022-03-17T20:09:30.4546889Z ---------------------------------------------------------------------- 2022-03-17T20:09:30.4911007Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106217 2022-03-17T20:09:30.5015616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106218 2022-03-17T20:09:31.3064097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqrl7ux_s 2022-03-17T20:09:31.3065286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqrl7ux_s/_remote_module_non_sriptable.py 2022-03-17T20:09:31.3467236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14mugk94 2022-03-17T20:09:31.3469626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14mugk94/_remote_module_non_sriptable.py 2022-03-17T20:09:32.9036768Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:32.9084019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:32.9155319Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:32.9201021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:32.9313053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:32.9313767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:32.9314779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:32.9315521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:34.3103523Z ok (3.855s) 2022-03-17T20:09:34.3103796Z 2022-03-17T20:09:34.3104423Z ---------------------------------------------------------------------- 2022-03-17T20:09:34.3104752Z Ran 1 test in 3.856s 2022-03-17T20:09:34.3104932Z 2022-03-17T20:09:34.3105006Z OK 2022-03-17T20:09:34.3105122Z 2022-03-17T20:09:34.3106681Z Generating XML reports... 2022-03-17T20:09:34.3162276Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200930.xml 2022-03-17T20:09:35.3606921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03ena_rs 2022-03-17T20:09:35.3607777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03ena_rs/_remote_module_non_sriptable.py 2022-03-17T20:09:36.8947080Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:36.9014263Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:36.9030116Z 2022-03-17T20:09:36.9030573Z Running tests... 2022-03-17T20:09:36.9031061Z ---------------------------------------------------------------------- 2022-03-17T20:09:36.9393609Z test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106339 2022-03-17T20:09:36.9497732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106340 2022-03-17T20:09:37.7335992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f36obpz 2022-03-17T20:09:37.7336882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f36obpz/_remote_module_non_sriptable.py 2022-03-17T20:09:37.7620114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ovgfl6p 2022-03-17T20:09:37.7621181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ovgfl6p/_remote_module_non_sriptable.py 2022-03-17T20:09:39.3280708Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:39.3326108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:39.3327162Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:39.3372672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:39.3537396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:39.3537929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:39.3538699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:39.3539388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:40.7583765Z ok (3.855s) 2022-03-17T20:09:40.7583969Z 2022-03-17T20:09:40.7584611Z ---------------------------------------------------------------------- 2022-03-17T20:09:40.7584976Z Ran 1 test in 3.855s 2022-03-17T20:09:40.7585127Z 2022-03-17T20:09:40.7585223Z OK 2022-03-17T20:09:40.7585337Z 2022-03-17T20:09:40.7585521Z Generating XML reports... 2022-03-17T20:09:40.7641939Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200936.xml 2022-03-17T20:09:41.8090568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ux9b1fr 2022-03-17T20:09:41.8091430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ux9b1fr/_remote_module_non_sriptable.py 2022-03-17T20:09:43.3292918Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:43.3359045Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:43.3374897Z 2022-03-17T20:09:43.3375084Z Running tests... 2022-03-17T20:09:43.3375519Z ---------------------------------------------------------------------- 2022-03-17T20:09:43.3740618Z test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106461 2022-03-17T20:09:43.3845844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106462 2022-03-17T20:09:44.1584031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftxj9img 2022-03-17T20:09:44.1586659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftxj9img/_remote_module_non_sriptable.py 2022-03-17T20:09:44.1911787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58_koepe 2022-03-17T20:09:44.1912959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58_koepe/_remote_module_non_sriptable.py 2022-03-17T20:09:45.7369149Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:45.7414111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:45.7550787Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:45.7595416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:45.7806745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:45.7807485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:45.7808271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:45.7808961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:47.1933139Z ok (3.855s) 2022-03-17T20:09:47.1933361Z 2022-03-17T20:09:47.1933801Z ---------------------------------------------------------------------- 2022-03-17T20:09:47.1934067Z Ran 1 test in 3.856s 2022-03-17T20:09:47.1934238Z 2022-03-17T20:09:47.1934328Z OK 2022-03-17T20:09:47.1934460Z 2022-03-17T20:09:47.1934591Z Generating XML reports... 2022-03-17T20:09:47.1990665Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200943.xml 2022-03-17T20:09:48.2295000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8jvdge4 2022-03-17T20:09:48.2295835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8jvdge4/_remote_module_non_sriptable.py 2022-03-17T20:09:49.7598645Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:49.7666394Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:49.7681935Z 2022-03-17T20:09:49.7682121Z Running tests... 2022-03-17T20:09:49.7682836Z ---------------------------------------------------------------------- 2022-03-17T20:09:49.8062289Z test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106583 2022-03-17T20:09:49.8166998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106584 2022-03-17T20:09:50.6700722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8pvg0lc 2022-03-17T20:09:50.6701306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpql0xe74f 2022-03-17T20:09:50.6701908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8pvg0lc/_remote_module_non_sriptable.py 2022-03-17T20:09:50.6702438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpql0xe74f/_remote_module_non_sriptable.py 2022-03-17T20:09:52.2755854Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:52.2801483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:52.2847523Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:52.2892932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:52.3012183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:52.3012699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:52.3013497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:52.3014196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:53.7254554Z ok (3.957s) 2022-03-17T20:09:53.7254794Z 2022-03-17T20:09:53.7255208Z ---------------------------------------------------------------------- 2022-03-17T20:09:53.7255829Z Ran 1 test in 3.957s 2022-03-17T20:09:53.7255995Z 2022-03-17T20:09:53.7256088Z OK 2022-03-17T20:09:53.7256225Z 2022-03-17T20:09:53.7256345Z Generating XML reports... 2022-03-17T20:09:53.7311340Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200949.xml 2022-03-17T20:09:54.7550627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeke7h5yg 2022-03-17T20:09:54.7555230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeke7h5yg/_remote_module_non_sriptable.py 2022-03-17T20:09:56.2852702Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:56.2920361Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:09:56.2935284Z 2022-03-17T20:09:56.2935667Z Running tests... 2022-03-17T20:09:56.2936130Z ---------------------------------------------------------------------- 2022-03-17T20:09:56.3297268Z test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106705 2022-03-17T20:09:56.3403244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106706 2022-03-17T20:09:57.1494479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p8_lo6s 2022-03-17T20:09:57.1495725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p8_lo6s/_remote_module_non_sriptable.py 2022-03-17T20:09:57.1820417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74y45gor 2022-03-17T20:09:57.1822140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74y45gor/_remote_module_non_sriptable.py 2022-03-17T20:09:58.7444971Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:58.7445262Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:09:58.7491089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:09:58.7491527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:09:58.7601927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:09:58.7605598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:09:58.7606706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:09:58.7704947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:00.1487690Z ok (3.855s) 2022-03-17T20:10:00.1488081Z 2022-03-17T20:10:00.1488547Z ---------------------------------------------------------------------- 2022-03-17T20:10:00.1488814Z Ran 1 test in 3.855s 2022-03-17T20:10:00.1488976Z 2022-03-17T20:10:00.1489049Z OK 2022-03-17T20:10:00.1489185Z 2022-03-17T20:10:00.1489325Z Generating XML reports... 2022-03-17T20:10:00.1545008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200956.xml 2022-03-17T20:10:01.2040083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubjj76z0 2022-03-17T20:10:01.2041051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubjj76z0/_remote_module_non_sriptable.py 2022-03-17T20:10:02.7576815Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:02.7642671Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo 2022-03-17T20:10:02.7658356Z 2022-03-17T20:10:02.7658914Z Running tests... 2022-03-17T20:10:02.7659682Z ---------------------------------------------------------------------- 2022-03-17T20:10:02.8041697Z test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106827 2022-03-17T20:10:02.8148639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106828 2022-03-17T20:10:03.6325826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2tbtnlz 2022-03-17T20:10:03.6326735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2tbtnlz/_remote_module_non_sriptable.py 2022-03-17T20:10:03.6507999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklrd5stq 2022-03-17T20:10:03.6509686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklrd5stq/_remote_module_non_sriptable.py 2022-03-17T20:10:05.2444130Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:05.2486411Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:05.2490591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:10:05.2530892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:10:05.2743775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:10:05.2744331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:10:05.2745358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:05.2746037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:06.6231617Z ok (3.857s) 2022-03-17T20:10:06.6231814Z 2022-03-17T20:10:06.6232192Z ---------------------------------------------------------------------- 2022-03-17T20:10:06.6232556Z Ran 1 test in 3.857s 2022-03-17T20:10:06.6232740Z 2022-03-17T20:10:06.6232831Z OK 2022-03-17T20:10:06.6232954Z 2022-03-17T20:10:06.6233086Z Generating XML reports... 2022-03-17T20:10:06.6287992Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317201002.xml 2022-03-17T20:10:07.0883604Z Running distributed/test_c10d_spawn_nccl ... [2022-03-17 20:10:07.087863] 2022-03-17T20:10:07.0884293Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:10:07.087973] 2022-03-17T20:10:07.8939963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmcv3ch1 2022-03-17T20:10:07.8941170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmcv3ch1/_remote_module_non_sriptable.py 2022-03-17T20:10:09.4344748Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:09.4410216Z , <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]> 2022-03-17T20:10:09.4411803Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:10:09.4412531Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:10:09.4412969Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:10:09.4413351Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) 2022-03-17T20:10:09.4413728Z 2022-03-17T20:10:09.4414016Z 2022-03-17T20:10:09.4415117Z , <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>]> 2022-03-17T20:10:09.4416435Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:10:09.4416815Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:10:09.4417214Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:10:09.4417619Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:10:09.4417995Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:10:09.4418385Z test_reduce (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:10:09.4418764Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) 2022-03-17T20:10:10.2417694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpon4qwyus 2022-03-17T20:10:10.2418967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpon4qwyus/_remote_module_non_sriptable.py 2022-03-17T20:10:11.7923340Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:11.7991164Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:10:11.8007033Z 2022-03-17T20:10:11.8007309Z Running tests... 2022-03-17T20:10:11.8007803Z ---------------------------------------------------------------------- 2022-03-17T20:10:13.6383836Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1alc7p4a 2022-03-17T20:10:13.6385184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1alc7p4a/_remote_module_non_sriptable.py 2022-03-17T20:10:13.6891227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4rbxnfa 2022-03-17T20:10:13.6892305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4rbxnfa/_remote_module_non_sriptable.py 2022-03-17T20:10:15.2415510Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:15.2416214Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:16.7962692Z ok (4.995s) 2022-03-17T20:10:16.7963245Z 2022-03-17T20:10:16.7963838Z ---------------------------------------------------------------------- 2022-03-17T20:10:16.7964156Z Ran 1 test in 4.996s 2022-03-17T20:10:16.7964306Z 2022-03-17T20:10:16.7964398Z OK 2022-03-17T20:10:16.7964540Z 2022-03-17T20:10:16.7964626Z Generating XML reports... 2022-03-17T20:10:16.8018370Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317201011.xml 2022-03-17T20:10:17.8940281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8kcwttk 2022-03-17T20:10:17.8941160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8kcwttk/_remote_module_non_sriptable.py 2022-03-17T20:10:19.4405472Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:19.4471970Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:10:19.4487488Z 2022-03-17T20:10:19.4487907Z Running tests... 2022-03-17T20:10:19.4488426Z ---------------------------------------------------------------------- 2022-03-17T20:10:21.3179802Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqy41fc8 2022-03-17T20:10:21.3181176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqy41fc8/_remote_module_non_sriptable.py 2022-03-17T20:10:21.3489388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3cptcfw 2022-03-17T20:10:21.3491432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3cptcfw/_remote_module_non_sriptable.py 2022-03-17T20:10:22.9317253Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:22.9414515Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:24.4676184Z ok (5.019s) 2022-03-17T20:10:24.4676734Z 2022-03-17T20:10:24.4677130Z ---------------------------------------------------------------------- 2022-03-17T20:10:24.4677444Z Ran 1 test in 5.019s 2022-03-17T20:10:24.4677607Z 2022-03-17T20:10:24.4677707Z OK 2022-03-17T20:10:24.4678114Z 2022-03-17T20:10:24.4678206Z Generating XML reports... 2022-03-17T20:10:24.4732978Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317201019.xml 2022-03-17T20:10:25.5719741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5j7002lb 2022-03-17T20:10:25.5720775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5j7002lb/_remote_module_non_sriptable.py 2022-03-17T20:10:27.1210568Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:27.1276857Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:10:27.1292101Z 2022-03-17T20:10:27.1292791Z Running tests... 2022-03-17T20:10:27.1293276Z ---------------------------------------------------------------------- 2022-03-17T20:10:28.9875379Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhcdg32v 2022-03-17T20:10:28.9876440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhcdg32v/_remote_module_non_sriptable.py 2022-03-17T20:10:29.0268655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmprps1ig 2022-03-17T20:10:29.0271754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmprps1ig/_remote_module_non_sriptable.py 2022-03-17T20:10:30.5745757Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:30.5863694Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:32.1161963Z ok (4.987s) 2022-03-17T20:10:32.1162203Z 2022-03-17T20:10:32.1162661Z ---------------------------------------------------------------------- 2022-03-17T20:10:32.1162973Z Ran 1 test in 4.987s 2022-03-17T20:10:32.1163156Z 2022-03-17T20:10:32.1163252Z OK 2022-03-17T20:10:32.1165553Z 2022-03-17T20:10:32.1165954Z Generating XML reports... 2022-03-17T20:10:32.1218199Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317201027.xml 2022-03-17T20:10:33.2306290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23zehq9a 2022-03-17T20:10:33.2307275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23zehq9a/_remote_module_non_sriptable.py 2022-03-17T20:10:34.7678431Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:34.7744612Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:10:34.7760050Z 2022-03-17T20:10:34.7760171Z Running tests... 2022-03-17T20:10:34.7761155Z ---------------------------------------------------------------------- 2022-03-17T20:10:36.6557454Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhq4l6pq 2022-03-17T20:10:36.6558564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhq4l6pq/_remote_module_non_sriptable.py 2022-03-17T20:10:36.6606511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdkyfqg8 2022-03-17T20:10:36.6609085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdkyfqg8/_remote_module_non_sriptable.py 2022-03-17T20:10:38.2651714Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:38.2686830Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:39.7939395Z ok (5.018s) 2022-03-17T20:10:39.7940677Z 2022-03-17T20:10:39.7941098Z ---------------------------------------------------------------------- 2022-03-17T20:10:39.7941478Z Ran 1 test in 5.018s 2022-03-17T20:10:39.7941693Z 2022-03-17T20:10:39.7941724Z OK 2022-03-17T20:10:39.7941835Z 2022-03-17T20:10:39.7941964Z Generating XML reports... 2022-03-17T20:10:39.7995157Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317201034.xml 2022-03-17T20:10:40.8849615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52yxs0h6 2022-03-17T20:10:40.8850970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52yxs0h6/_remote_module_non_sriptable.py 2022-03-17T20:10:42.4465164Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:42.4531477Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:10:42.4547539Z 2022-03-17T20:10:42.4547908Z Running tests... 2022-03-17T20:10:42.4548393Z ---------------------------------------------------------------------- 2022-03-17T20:10:42.4911780Z test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107489 2022-03-17T20:10:42.5018836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107490 2022-03-17T20:10:43.3062441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71pyae_j 2022-03-17T20:10:43.3063581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71pyae_j/_remote_module_non_sriptable.py 2022-03-17T20:10:43.3467132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pi_a4eg 2022-03-17T20:10:43.3469273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pi_a4eg/_remote_module_non_sriptable.py 2022-03-17T20:10:44.8991243Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:44.9036943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:10:44.9040584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:10:44.9081276Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:44.9126364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:10:44.9130092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:10:44.9131324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:44.9143671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:46.3108675Z ok (3.856s) 2022-03-17T20:10:46.3108877Z 2022-03-17T20:10:46.3109546Z ---------------------------------------------------------------------- 2022-03-17T20:10:46.3109891Z Ran 1 test in 3.856s 2022-03-17T20:10:46.3110044Z 2022-03-17T20:10:46.3110062Z OK 2022-03-17T20:10:46.3110202Z 2022-03-17T20:10:46.3110380Z Generating XML reports... 2022-03-17T20:10:46.3165113Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201042.xml 2022-03-17T20:10:47.3717803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rywy4ze 2022-03-17T20:10:47.3719514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rywy4ze/_remote_module_non_sriptable.py 2022-03-17T20:10:48.9173509Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:48.9240216Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:10:48.9255908Z 2022-03-17T20:10:48.9256208Z Running tests... 2022-03-17T20:10:48.9256646Z ---------------------------------------------------------------------- 2022-03-17T20:10:48.9620078Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107614 2022-03-17T20:10:48.9727468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107615 2022-03-17T20:10:49.7657109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3anup0jg 2022-03-17T20:10:49.7658004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3anup0jg/_remote_module_non_sriptable.py 2022-03-17T20:10:49.7851549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1pcx8zu 2022-03-17T20:10:49.7853210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1pcx8zu/_remote_module_non_sriptable.py 2022-03-17T20:10:51.3634801Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:51.3680434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:10:51.3684178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:10:51.3837026Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:51.3881972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:10:51.3885518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:10:51.3886806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:51.3888047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:52.7830297Z ok (3.857s) 2022-03-17T20:10:52.7830609Z 2022-03-17T20:10:52.7831205Z ---------------------------------------------------------------------- 2022-03-17T20:10:52.7833883Z Ran 1 test in 3.857s 2022-03-17T20:10:52.7834183Z 2022-03-17T20:10:52.7834348Z OK 2022-03-17T20:10:52.7834644Z 2022-03-17T20:10:52.7834915Z Generating XML reports... 2022-03-17T20:10:52.7887050Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201048.xml 2022-03-17T20:10:53.8418655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9y6me9uz 2022-03-17T20:10:53.8419706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9y6me9uz/_remote_module_non_sriptable.py 2022-03-17T20:10:55.3657238Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:55.3723969Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:10:55.3739523Z 2022-03-17T20:10:55.3739644Z Running tests... 2022-03-17T20:10:55.3740359Z ---------------------------------------------------------------------- 2022-03-17T20:10:55.4102427Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107739 2022-03-17T20:10:55.4208327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107740 2022-03-17T20:10:56.2233490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g8b3go8 2022-03-17T20:10:56.2234620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g8b3go8/_remote_module_non_sriptable.py 2022-03-17T20:10:56.2328479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpica324nv 2022-03-17T20:10:56.2330970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpica324nv/_remote_module_non_sriptable.py 2022-03-17T20:10:57.8259574Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:57.8305321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:10:57.8309365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:10:57.8434609Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:10:57.8479600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:10:57.8483353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:10:57.8484423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:57.8513476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:10:59.3296786Z ok (3.955s) 2022-03-17T20:10:59.3297013Z 2022-03-17T20:10:59.3297408Z ---------------------------------------------------------------------- 2022-03-17T20:10:59.3297763Z Ran 1 test in 3.956s 2022-03-17T20:10:59.3297919Z 2022-03-17T20:10:59.3298257Z OK 2022-03-17T20:10:59.3298407Z 2022-03-17T20:10:59.3298542Z Generating XML reports... 2022-03-17T20:10:59.3358543Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201055.xml 2022-03-17T20:11:00.3862379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnazvzk8u 2022-03-17T20:11:00.3864733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnazvzk8u/_remote_module_non_sriptable.py 2022-03-17T20:11:01.8967040Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:01.9033616Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:11:01.9049870Z 2022-03-17T20:11:01.9050020Z Running tests... 2022-03-17T20:11:01.9050814Z ---------------------------------------------------------------------- 2022-03-17T20:11:01.9417832Z test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107864 2022-03-17T20:11:01.9525576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107865 2022-03-17T20:11:02.7601019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply9ml6fw 2022-03-17T20:11:02.7602055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply9ml6fw/_remote_module_non_sriptable.py 2022-03-17T20:11:02.7736258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd64obf54 2022-03-17T20:11:02.7739015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd64obf54/_remote_module_non_sriptable.py 2022-03-17T20:11:04.3697830Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:04.3743380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:04.3747395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:04.3818096Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:04.3863917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:04.3867781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:04.3869028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:04.3951876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:05.7613221Z ok (3.856s) 2022-03-17T20:11:05.7613438Z 2022-03-17T20:11:05.7613785Z ---------------------------------------------------------------------- 2022-03-17T20:11:05.7614174Z Ran 1 test in 3.856s 2022-03-17T20:11:05.7614352Z 2022-03-17T20:11:05.7614447Z OK 2022-03-17T20:11:05.7614566Z 2022-03-17T20:11:05.7614694Z Generating XML reports... 2022-03-17T20:11:05.7670140Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201101.xml 2022-03-17T20:11:06.8207329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tjzp22a 2022-03-17T20:11:06.8208210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tjzp22a/_remote_module_non_sriptable.py 2022-03-17T20:11:08.3631277Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:08.3694251Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:11:08.3710110Z 2022-03-17T20:11:08.3710593Z Running tests... 2022-03-17T20:11:08.3711077Z ---------------------------------------------------------------------- 2022-03-17T20:11:08.4072711Z test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107989 2022-03-17T20:11:08.4177234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107990 2022-03-17T20:11:09.2148845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55b9whyg 2022-03-17T20:11:09.2150018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55b9whyg/_remote_module_non_sriptable.py 2022-03-17T20:11:09.2362190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgel8gye 2022-03-17T20:11:09.2363531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgel8gye/_remote_module_non_sriptable.py 2022-03-17T20:11:10.8178538Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:10.8223361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:10.8228079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:10.8294806Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:10.8338834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:10.8342478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:10.8343665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:10.8432249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:12.2278572Z ok (3.856s) 2022-03-17T20:11:12.2278775Z 2022-03-17T20:11:12.2279180Z ---------------------------------------------------------------------- 2022-03-17T20:11:12.2279498Z Ran 1 test in 3.857s 2022-03-17T20:11:12.2279662Z 2022-03-17T20:11:12.2279754Z OK 2022-03-17T20:11:12.2279895Z 2022-03-17T20:11:12.2279997Z Generating XML reports... 2022-03-17T20:11:12.2333864Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201108.xml 2022-03-17T20:11:13.2631924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhjoil3v 2022-03-17T20:11:13.2633072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhjoil3v/_remote_module_non_sriptable.py 2022-03-17T20:11:14.7849297Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:14.7914774Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:11:14.7930072Z 2022-03-17T20:11:14.7930494Z Running tests... 2022-03-17T20:11:14.7930926Z ---------------------------------------------------------------------- 2022-03-17T20:11:14.8292540Z test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108114 2022-03-17T20:11:14.8397852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108115 2022-03-17T20:11:15.6485452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyv3texo 2022-03-17T20:11:15.6486453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyv3texo/_remote_module_non_sriptable.py 2022-03-17T20:11:15.6882320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjea1v5a 2022-03-17T20:11:15.6883704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjea1v5a/_remote_module_non_sriptable.py 2022-03-17T20:11:17.2373007Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:17.2418745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:17.2422738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:17.2648045Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:17.2693008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:17.2696866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:17.2697661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:17.2728780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:18.6499810Z ok (3.857s) 2022-03-17T20:11:18.6500025Z 2022-03-17T20:11:18.6500647Z ---------------------------------------------------------------------- 2022-03-17T20:11:18.6500948Z Ran 1 test in 3.857s 2022-03-17T20:11:18.6501107Z 2022-03-17T20:11:18.6501180Z OK 2022-03-17T20:11:18.6501314Z 2022-03-17T20:11:18.6501446Z Generating XML reports... 2022-03-17T20:11:18.6555758Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201114.xml 2022-03-17T20:11:19.7098105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x79wnab 2022-03-17T20:11:19.7098904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x79wnab/_remote_module_non_sriptable.py 2022-03-17T20:11:21.2455654Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:21.2530102Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl 2022-03-17T20:11:21.2545748Z 2022-03-17T20:11:21.2546153Z Running tests... 2022-03-17T20:11:21.2546659Z ---------------------------------------------------------------------- 2022-03-17T20:11:21.2923803Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108239 2022-03-17T20:11:21.3028554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108240 2022-03-17T20:11:22.0839584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1uguaza 2022-03-17T20:11:22.0840498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1uguaza/_remote_module_non_sriptable.py 2022-03-17T20:11:22.1123714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7z38uwl 2022-03-17T20:11:22.1125193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7z38uwl/_remote_module_non_sriptable.py 2022-03-17T20:11:23.6829755Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:23.6854098Z INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:23.6875772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:23.6880465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:23.6897678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:23.6901626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:23.6902682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:23.6984194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:25.1116340Z ok (3.857s) 2022-03-17T20:11:25.1116543Z 2022-03-17T20:11:25.1116981Z ---------------------------------------------------------------------- 2022-03-17T20:11:25.1117317Z Ran 1 test in 3.857s 2022-03-17T20:11:25.1117483Z 2022-03-17T20:11:25.1117508Z OK 2022-03-17T20:11:25.1117618Z 2022-03-17T20:11:25.1117759Z Generating XML reports... 2022-03-17T20:11:25.1172570Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201121.xml 2022-03-17T20:11:25.5695377Z Running distributed/test_data_parallel ... [2022-03-17 20:11:25.569026] 2022-03-17T20:11:25.5696086Z Executing ['/opt/conda/bin/python', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:11:25.569133] 2022-03-17T20:11:27.9177428Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel 2022-03-17T20:11:27.9201331Z 2022-03-17T20:11:27.9201666Z Running tests... 2022-03-17T20:11:27.9202113Z ---------------------------------------------------------------------- 2022-03-17T20:11:29.4093051Z test_autocast (__main__.TestDataParallel) ... ok (1.489s) 2022-03-17T20:11:29.5129360Z test_data_parallel (__main__.TestDataParallel) ... ok (0.104s) 2022-03-17T20:11:29.5250452Z test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... ok (0.012s) 2022-03-17T20:11:29.5280186Z test_data_parallel_complex (__main__.TestDataParallel) ... ok (0.003s) 2022-03-17T20:11:29.5339773Z test_data_parallel_device_args (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:11:29.5399369Z test_data_parallel_function_deletion (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:11:29.5413092Z test_data_parallel_lazy_linear (__main__.TestDataParallel) ... /opt/conda/lib/python3.7/site-packages/torch/nn/modules/lazy.py:178: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. 2022-03-17T20:11:29.5413871Z warnings.warn('Lazy modules are a new feature under heavy development ' 2022-03-17T20:11:29.5422389Z ok (0.002s) 2022-03-17T20:11:29.5466631Z test_data_parallel_model_device (__main__.TestDataParallel) 2022-03-17T20:11:29.5758603Z Test device[0] check at forward time. ... ok (0.033s) 2022-03-17T20:11:29.6175875Z test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... ok (0.042s) 2022-03-17T20:11:29.6222382Z test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... ok (0.005s) 2022-03-17T20:11:29.6283610Z test_data_parallel_multiple_input (__main__.TestDataParallel) ... /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/comm.py:232: UserWarning: Using -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu". 2022-03-17T20:11:29.6284272Z 'Using -1 to represent CPU tensor is deprecated. Please use a ' 2022-03-17T20:11:29.6444380Z ok (0.022s) 2022-03-17T20:11:29.6475328Z test_data_parallel_nested_input (__main__.TestDataParallel) ... ok (0.003s) 2022-03-17T20:11:29.6537678Z test_data_parallel_nested_output (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:11:29.6578512Z test_data_parallel_no_grad (__main__.TestDataParallel) ... ok (0.004s) 2022-03-17T20:11:30.9693437Z test_data_parallel_rnn (__main__.TestDataParallel) ... ok (1.311s) 2022-03-17T20:11:30.9726378Z test_data_parallel_small_back (__main__.TestDataParallel) ... ok (0.003s) 2022-03-17T20:11:30.9845174Z test_data_parallel_sparse (__main__.TestDataParallel) ... ok (0.012s) 2022-03-17T20:11:31.0071946Z test_gather_cpu (__main__.TestDataParallel) ... /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. 2022-03-17T20:11:31.0072655Z warnings.warn('Was asked to gather along dimension 0, but all ' 2022-03-17T20:11:31.0288624Z ok (0.044s) 2022-03-17T20:11:31.0300908Z test_gather_different_len_dicts (__main__.TestDataParallel) ... ok (0.001s) 2022-03-17T20:11:31.0741748Z test_gather_gpu (__main__.TestDataParallel) ... ok (0.044s) 2022-03-17T20:11:31.0795598Z test_parallel_apply (__main__.TestDataParallel) ... ok (0.005s) 2022-03-17T20:11:31.0852846Z test_parallel_apply_autocast (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:11:31.0874462Z test_parallel_apply_passes_exception (__main__.TestDataParallel) ... ok (0.002s) 2022-03-17T20:11:31.0952011Z test_parameter_list_dict_replica (__main__.TestDataParallel) ... ok (0.008s) 2022-03-17T20:11:31.0995833Z test_replicate (__main__.TestDataParallel) ... ok (0.004s) 2022-03-17T20:11:31.1031223Z test_replicate_buffers (__main__.TestDataParallel) ... ok (0.003s) 2022-03-17T20:11:31.1068524Z test_save_replica_module (__main__.TestDataParallel) ... ok (0.004s) 2022-03-17T20:11:31.1252148Z test_scatter_cpu (__main__.TestDataParallel) ... ok (0.018s) 2022-03-17T20:11:31.1440497Z test_scatter_gpu (__main__.TestDataParallel) ... ok (0.019s) 2022-03-17T20:11:31.9638561Z test_strided_grad_layout (__main__.TestDataParallel) ... ok (0.819s) 2022-03-17T20:11:31.9696432Z test_zero_grad (__main__.TestDataParallel) ... ok (0.006s) 2022-03-17T20:11:31.9756567Z test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.006s) 2022-03-17T20:11:31.9809917Z test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2022-03-17T20:11:31.9860307Z test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s) 2022-03-17T20:11:32.1172268Z test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.131s) 2022-03-17T20:11:32.1413507Z test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.024s) 2022-03-17T20:11:32.1647233Z test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.1882235Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.2114500Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.2346507Z test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.2579495Z test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.2594574Z test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... skip (0.001s) 2022-03-17T20:11:32.2828743Z test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.3062369Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.3294827Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.3527577Z test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.023s) 2022-03-17T20:11:32.3527982Z 2022-03-17T20:11:32.3528372Z ---------------------------------------------------------------------- 2022-03-17T20:11:32.3528696Z Ran 46 tests in 4.433s 2022-03-17T20:11:32.3528809Z 2022-03-17T20:11:32.3528905Z OK (skipped=1) 2022-03-17T20:11:32.3529060Z 2022-03-17T20:11:32.3529186Z Generating XML reports... 2022-03-17T20:11:32.3595885Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220317201127.xml 2022-03-17T20:11:32.3614134Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20220317201127.xml 2022-03-17T20:11:32.9942480Z Running distributed/test_distributed_spawn ... [2022-03-17 20:11:32.993748] 2022-03-17T20:11:32.9984763Z /usr/bin/mpiexec 2022-03-17T20:11:32.9992553Z Running distributed tests for the test backend with env init_method 2022-03-17T20:11:32.9995000Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:11:32.999194] 2022-03-17T20:11:34.0436022Z 2022-03-17T20:11:34.1377064Z Running distributed tests for the test backend with file init_method 2022-03-17T20:11:34.1378828Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:11:34.137492] 2022-03-17T20:11:35.1648577Z 2022-03-17T20:11:35.2606511Z Running distributed tests for the mpi backend with env init_method 2022-03-17T20:11:35.3710424Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:11:35.370578] 2022-03-17T20:11:35.3809586Z Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/X3ICQQVZBVRFJAXET5XR67GUPC:/var/lib/docker/overlay2/l/CA7JFOG7ACAOEFVLM5MFLIYAIP:/var/lib/docker/overlay2/l/IGBJO43VPWN7RLYSM622IRV4F5:/var/lib/docker/overlay2/l/UITL5J4NPM3WIMHCPSIVI326LR:/var/lib/docker/overlay2/l/HQUPCLAEQ3YEQXFHNCNJ3QW3XS:/var/lib/docker/overlay2/l/HYL5UOE4SU5TDVV2AK3G5NOYOI:/var/lib/docker/overlay2/l/RDC6CYXRLSZBF7YRFT74JJYCNN:/var/lib/docker/overlay2/l/BDCCLXXICGOJ7FM4OEICE7DDUB:/var/lib/docker/overlay2/l/WBHPXCP6W772S' 2022-03-17T20:11:35.3811519Z Unexpected end of /proc/mounts line `QIXS3HH7SSJ6O:/var/lib/docker/overlay2/l/7DBWL7AVQA7SO25RU7PTGPPZL2:/var/lib/docker/overlay2/l/SNM4BSX7YCIDVDE4ETE57Q5VNJ:/var/lib/docker/overlay2/l/75HYOPEHGY53LU5K4X7Z3J6EVN:/var/lib/docker/overlay2/l/QAC34UQBXDBIQXLQP6DA2HW7G7:/var/lib/docker/overlay2/l/UF5K7NUYQFIOABVTSQ7ENUFLIE:/var/lib/docker/overlay2/l/MPKO7WWLDQEGYPTAJS5X6BAWCD:/var/lib/docker/overlay2/l/K6EIOLSYI4KBYIAATWI5G32DW6:/var/lib/docker/overlay2/l/7MCNVY3IEHBBGFDLTCMUNL2YP4:/var/lib/docker/overlay2/l/FNEOEEAJTRLDCPQG2TEKB4X6FY:/var/lib/do' 2022-03-17T20:11:35.3813277Z Unexpected end of /proc/mounts line `cker/overlay2/l/WNLAU2PQ2DEQ2H4QZ33PICCHPO:/var/lib/docker/overlay2/l/FUIBG5GAYPVRZ7PKLLV4HV7YGH:/var/lib/docker/overlay2/l/76FTSSKEP446V3DB6NCBMNMFMS:/var/lib/docker/overlay2/l/FVR6R3YS2ILSQH5JNHKQYNMGN7:/var/lib/docker/overlay2/l/JLVKWNTJ4T5Q6XGNWFCOFAAN6D:/var/lib/docker/overlay2/l/W4IHGRDDOQ2B7IXMIQN3EJRCOW:/var/lib/docker/overlay2/l/2JESUM7LOX5O7Z6ISHKZUN2OKO:/var/lib/docker/overlay2/l/5TSFUTN2QXFGJIQHPHL4HFXOJ4:/var/lib/docker/overlay2/l/CG6MRML2GIOSRUR5FRJKG32KFB:/var/lib/docker/overlay2/l/VUI4XPVYT' 2022-03-17T20:11:35.3815004Z Unexpected end of /proc/mounts line `DDAOZYOSFNFEYA5EH:/var/lib/docker/overlay2/l/QQ62OGZS3MWSMNGCAN2IU4VXPA:/var/lib/docker/overlay2/l/CFK5NRPKSQJIHFEZBHXCDC3GNF:/var/lib/docker/overlay2/l/VOHXHSSVVI7L3EO6DPUNJT75CY:/var/lib/docker/overlay2/l/XVYLBZ73K6D5YZWROKWYQSUBHD:/var/lib/docker/overlay2/l/HSGEIPMGDINHAJXDICTZMW5FPY:/var/lib/docker/overlay2/l/5HIG6ER3YJVQHLSFJOY4KHHK3F:/var/lib/docker/overlay2/l/BXEV75IV63ZTQUFTCEHADFMSLM:/var/lib/docker/overlay2/l/4JX5SVBVBG7E32SLVCBPGEADWG:/var/lib/docker/overlay2/l/R4IZ3CTW7ISJ4NI6D5HQJOAOMN:/var/li' 2022-03-17T20:11:35.3816752Z Unexpected end of /proc/mounts line `b/docker/overlay2/l/O56PFIC6QYLNIU6EEJOUHQRKZ2:/var/lib/docker/overlay2/l/AA3GSRXV3KAXH6RGBEDG6YQ64V:/var/lib/docker/overlay2/l/P6RU3HE63MNRBZP7R5GUTJGFV4:/var/lib/docker/overlay2/l/OYBD77M3QV5VF6JZC2T4HZ77VN:/var/lib/docker/overlay2/l/I34DXGHCWRUSQMVGSAALD5EPYU:/var/lib/docker/overlay2/l/K4UBAX4YLDSRX4AYRJXVNP7FEY:/var/lib/docker/overlay2/l/VQ54AYR7IIUM2TFSACGL6RYN2X:/var/lib/docker/overlay2/l/RJVBUDCBWDHJUATDEZQZ6Y3WZV:/var/lib/docker/overlay2/l/ZRHOTZCRKGAUID3M4CMBFORS6Q:/var/lib/docker/overlay2/l/3TKXK' 2022-03-17T20:11:36.4512005Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn 2022-03-17T20:11:36.4538572Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn 2022-03-17T20:11:36.4557834Z 2022-03-17T20:11:36.4558208Z Running tests... 2022-03-17T20:11:36.4558769Z ---------------------------------------------------------------------- 2022-03-17T20:11:36.4559030Z 2022-03-17T20:11:36.4559296Z ---------------------------------------------------------------------- 2022-03-17T20:11:36.4559614Z Ran 0 tests in 0.000s 2022-03-17T20:11:36.4559780Z 2022-03-17T20:11:36.4559875Z OK 2022-03-17T20:11:36.4560008Z 2022-03-17T20:11:36.4560119Z Generating XML reports... 2022-03-17T20:11:36.4585851Z 2022-03-17T20:11:36.4586105Z Running tests... 2022-03-17T20:11:36.4586539Z ---------------------------------------------------------------------- 2022-03-17T20:11:36.4586785Z 2022-03-17T20:11:36.4587027Z ---------------------------------------------------------------------- 2022-03-17T20:11:36.4587633Z Ran 0 tests in 0.000s 2022-03-17T20:11:36.4587798Z 2022-03-17T20:11:36.4587890Z OK 2022-03-17T20:11:36.4588005Z 2022-03-17T20:11:36.4588132Z Generating XML reports... 2022-03-17T20:11:36.4981987Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn 2022-03-17T20:11:36.5028650Z 2022-03-17T20:11:36.5029064Z Running tests... 2022-03-17T20:11:36.5029832Z ---------------------------------------------------------------------- 2022-03-17T20:11:36.5030073Z 2022-03-17T20:11:36.5030324Z ---------------------------------------------------------------------- 2022-03-17T20:11:36.5030613Z Ran 0 tests in 0.000s 2022-03-17T20:11:36.5030746Z 2022-03-17T20:11:36.5030841Z OK 2022-03-17T20:11:36.5031250Z 2022-03-17T20:11:36.5031398Z Generating XML reports... 2022-03-17T20:11:36.6191581Z Running distributed tests for the mpi backend with file init_method 2022-03-17T20:11:36.7258665Z Executing ['mpiexec', '-n', '3', '--noprefix', '--allow-run-as-root', '/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:11:36.725388] 2022-03-17T20:11:36.7359027Z Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/X3ICQQVZBVRFJAXET5XR67GUPC:/var/lib/docker/overlay2/l/CA7JFOG7ACAOEFVLM5MFLIYAIP:/var/lib/docker/overlay2/l/IGBJO43VPWN7RLYSM622IRV4F5:/var/lib/docker/overlay2/l/UITL5J4NPM3WIMHCPSIVI326LR:/var/lib/docker/overlay2/l/HQUPCLAEQ3YEQXFHNCNJ3QW3XS:/var/lib/docker/overlay2/l/HYL5UOE4SU5TDVV2AK3G5NOYOI:/var/lib/docker/overlay2/l/RDC6CYXRLSZBF7YRFT74JJYCNN:/var/lib/docker/overlay2/l/BDCCLXXICGOJ7FM4OEICE7DDUB:/var/lib/docker/overlay2/l/WBHPXCP6W772S' 2022-03-17T20:11:36.7360806Z Unexpected end of /proc/mounts line `QIXS3HH7SSJ6O:/var/lib/docker/overlay2/l/7DBWL7AVQA7SO25RU7PTGPPZL2:/var/lib/docker/overlay2/l/SNM4BSX7YCIDVDE4ETE57Q5VNJ:/var/lib/docker/overlay2/l/75HYOPEHGY53LU5K4X7Z3J6EVN:/var/lib/docker/overlay2/l/QAC34UQBXDBIQXLQP6DA2HW7G7:/var/lib/docker/overlay2/l/UF5K7NUYQFIOABVTSQ7ENUFLIE:/var/lib/docker/overlay2/l/MPKO7WWLDQEGYPTAJS5X6BAWCD:/var/lib/docker/overlay2/l/K6EIOLSYI4KBYIAATWI5G32DW6:/var/lib/docker/overlay2/l/7MCNVY3IEHBBGFDLTCMUNL2YP4:/var/lib/docker/overlay2/l/FNEOEEAJTRLDCPQG2TEKB4X6FY:/var/lib/do' 2022-03-17T20:11:36.7362541Z Unexpected end of /proc/mounts line `cker/overlay2/l/WNLAU2PQ2DEQ2H4QZ33PICCHPO:/var/lib/docker/overlay2/l/FUIBG5GAYPVRZ7PKLLV4HV7YGH:/var/lib/docker/overlay2/l/76FTSSKEP446V3DB6NCBMNMFMS:/var/lib/docker/overlay2/l/FVR6R3YS2ILSQH5JNHKQYNMGN7:/var/lib/docker/overlay2/l/JLVKWNTJ4T5Q6XGNWFCOFAAN6D:/var/lib/docker/overlay2/l/W4IHGRDDOQ2B7IXMIQN3EJRCOW:/var/lib/docker/overlay2/l/2JESUM7LOX5O7Z6ISHKZUN2OKO:/var/lib/docker/overlay2/l/5TSFUTN2QXFGJIQHPHL4HFXOJ4:/var/lib/docker/overlay2/l/CG6MRML2GIOSRUR5FRJKG32KFB:/var/lib/docker/overlay2/l/VUI4XPVYT' 2022-03-17T20:11:36.7364300Z Unexpected end of /proc/mounts line `DDAOZYOSFNFEYA5EH:/var/lib/docker/overlay2/l/QQ62OGZS3MWSMNGCAN2IU4VXPA:/var/lib/docker/overlay2/l/CFK5NRPKSQJIHFEZBHXCDC3GNF:/var/lib/docker/overlay2/l/VOHXHSSVVI7L3EO6DPUNJT75CY:/var/lib/docker/overlay2/l/XVYLBZ73K6D5YZWROKWYQSUBHD:/var/lib/docker/overlay2/l/HSGEIPMGDINHAJXDICTZMW5FPY:/var/lib/docker/overlay2/l/5HIG6ER3YJVQHLSFJOY4KHHK3F:/var/lib/docker/overlay2/l/BXEV75IV63ZTQUFTCEHADFMSLM:/var/lib/docker/overlay2/l/4JX5SVBVBG7E32SLVCBPGEADWG:/var/lib/docker/overlay2/l/R4IZ3CTW7ISJ4NI6D5HQJOAOMN:/var/li' 2022-03-17T20:11:36.7366032Z Unexpected end of /proc/mounts line `b/docker/overlay2/l/O56PFIC6QYLNIU6EEJOUHQRKZ2:/var/lib/docker/overlay2/l/AA3GSRXV3KAXH6RGBEDG6YQ64V:/var/lib/docker/overlay2/l/P6RU3HE63MNRBZP7R5GUTJGFV4:/var/lib/docker/overlay2/l/OYBD77M3QV5VF6JZC2T4HZ77VN:/var/lib/docker/overlay2/l/I34DXGHCWRUSQMVGSAALD5EPYU:/var/lib/docker/overlay2/l/K4UBAX4YLDSRX4AYRJXVNP7FEY:/var/lib/docker/overlay2/l/VQ54AYR7IIUM2TFSACGL6RYN2X:/var/lib/docker/overlay2/l/RJVBUDCBWDHJUATDEZQZ6Y3WZV:/var/lib/docker/overlay2/l/ZRHOTZCRKGAUID3M4CMBFORS6Q:/var/lib/docker/overlay2/l/3TKXK' 2022-03-17T20:11:37.7975548Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn 2022-03-17T20:11:37.7977148Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn 2022-03-17T20:11:37.8019777Z 2022-03-17T20:11:37.8020266Z Running tests... 2022-03-17T20:11:37.8021319Z ---------------------------------------------------------------------- 2022-03-17T20:11:37.8021759Z 2022-03-17T20:11:37.8022047Z ---------------------------------------------------------------------- 2022-03-17T20:11:37.8022592Z Ran 0 tests in 0.000s 2022-03-17T20:11:37.8022860Z 2022-03-17T20:11:37.8022964Z OK 2022-03-17T20:11:37.8023064Z 2022-03-17T20:11:37.8023200Z Generating XML reports... 2022-03-17T20:11:37.8023371Z 2022-03-17T20:11:37.8023724Z Running tests... 2022-03-17T20:11:37.8024441Z ---------------------------------------------------------------------- 2022-03-17T20:11:37.8024690Z 2022-03-17T20:11:37.8024955Z ---------------------------------------------------------------------- 2022-03-17T20:11:37.8025269Z Ran 0 tests in 0.000s 2022-03-17T20:11:37.8025431Z 2022-03-17T20:11:37.8025522Z OK 2022-03-17T20:11:37.8025657Z 2022-03-17T20:11:37.8025761Z Generating XML reports... 2022-03-17T20:11:37.8080506Z Test results will be stored in test-reports/dist-mpi/distributed.test_distributed_spawn 2022-03-17T20:11:37.8128422Z 2022-03-17T20:11:37.8128775Z Running tests... 2022-03-17T20:11:37.8129225Z ---------------------------------------------------------------------- 2022-03-17T20:11:37.8129519Z 2022-03-17T20:11:37.8129779Z ---------------------------------------------------------------------- 2022-03-17T20:11:37.8130087Z Ran 0 tests in 0.000s 2022-03-17T20:11:37.8130219Z 2022-03-17T20:11:37.8130312Z OK 2022-03-17T20:11:37.8130448Z 2022-03-17T20:11:37.8130592Z Generating XML reports... 2022-03-17T20:11:37.9343231Z Running distributed tests for the nccl backend with env init_method 2022-03-17T20:11:37.9344462Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:11:37.934129] 2022-03-17T20:11:38.9753142Z 2022-03-17T20:11:38.9794229Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_get_bucket_sizes>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_params_and_buffers>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-03-17T20:11:38.9827961Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9828466Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9828867Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9829277Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9829698Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9830164Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9830731Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9831234Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9831737Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9832258Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9832807Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9833321Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9833825Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9834334Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9834793Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9835274Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9835728Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9836137Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9836561Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9836985Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9837453Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9837916Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9838303Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9838692Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9839101Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9839519Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9839929Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9840328Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9840739Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9841102Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9841489Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9841882Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9842247Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9842632Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9843015Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9843430Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9843844Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9844248Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9844684Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9845195Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9845684Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9846104Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9846506Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9846930Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9847340Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9847820Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9848263Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9848677Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9849082Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9849475Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9849890Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9850291Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9850688Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9851098Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9851487Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9851889Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9852259Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9852659Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9853052Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9853412Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9853776Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9854139Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9854538Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9854937Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9855306Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9855683Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9856042Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9856433Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9856818Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9857190Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9857595Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9857966Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9858334Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9858690Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9859067Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9859457Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9859833Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9860604Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9861075Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9861474Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9861898Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9862311Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9862851Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9863280Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9863729Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9864169Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9864589Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9865014Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9865492Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9865936Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9866376Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9866814Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9867267Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9867695Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9868140Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9868556Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9868925Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9869306Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9869654Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9870015Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9870390Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9870766Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9871147Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9871508Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9871902Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9872300Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9872671Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9873060Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9873442Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9873865Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9874275Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9874665Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9875075Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9875463Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9875864Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9876270Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9876635Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9876999Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9877368Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9877750Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9878119Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9878510Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9878951Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9879506Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9879951Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9880340Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9880751Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9881172Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9881595Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9882103Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9882528Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9882946Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9883380Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9883772Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9884141Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9884509Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9884906Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9885302Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9885693Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9886118Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9886543Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9886954Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9887402Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9887880Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9888430Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9889037Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9889616Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9890208Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9890782Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9891380Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9891963Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9892538Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9893077Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9893570Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9893993Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9894373Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9894751Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9895224Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9895615Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9896015Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9896448Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9896884Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9897344Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9897753Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9898172Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9898590Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9899001Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9899430Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9899837Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9900603Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9901124Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9901535Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9901952Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9902364Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9902768Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9903178Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9903555Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9903968Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9904426Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9904834Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9905211Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9905588Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9905998Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9906408Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9906766Z test_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9907126Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9907478Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9907850Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9908208Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9908578Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9908959Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9909318Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9909676Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9910011Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9910382Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9910771Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9911139Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9911509Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9911836Z test_isend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9912209Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9912606Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9913150Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9913595Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9914019Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9914425Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9914836Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9915247Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9915731Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9916140Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9916551Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9916962Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9917350Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9917754Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9918125Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9918519Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9918961Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9919444Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9919915Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9920340Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9920790Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9921241Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9921648Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9922065Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9922463Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9922868Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9923258Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9923637Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9924037Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9924401Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9924773Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9925143Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9925529Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9925895Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9926237Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9926607Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9926962Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9927325Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9927690Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9928052Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9928434Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9928781Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9929140Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9929497Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9929937Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9930314Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9930684Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9931056Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9931417Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9931784Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9932154Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9932549Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9933042Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9933475Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9933847Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9934245Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9934652Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9935061Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9935432Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9935839Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9936265Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9936654Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9937056Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9937440Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9937831Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9938209Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9938608Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9939049Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:38.9939467Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:11:40.0230213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:40.0244644Z 2022-03-17T20:11:40.0244939Z Running tests... 2022-03-17T20:11:40.0245599Z ---------------------------------------------------------------------- 2022-03-17T20:11:41.5467834Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:41.5889022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108911 2022-03-17T20:11:41.6013782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108912 2022-03-17T20:11:42.6406891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:42.6459555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:42.6460786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:42.6508318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:42.6515026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:42.7475690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:43.9305985Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:11:43.9306957Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:11:44.0107133Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:11:44.0108025Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:11:45.0381535Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:11:45.0382111Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:11:45.0383342Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:11:45.0384306Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:11:45.0519945Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:11:45.0520797Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:11:45.0522334Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:11:45.0523157Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:11:45.0661785Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:11:45.0662640Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:11:45.0666303Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:11:45.0667179Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:11:45.4104804Z ok (5.386s) 2022-03-17T20:11:45.4105080Z 2022-03-17T20:11:45.4105465Z ---------------------------------------------------------------------- 2022-03-17T20:11:45.4105789Z Ran 1 test in 5.386s 2022-03-17T20:11:45.4105958Z 2022-03-17T20:11:45.4106078Z OK 2022-03-17T20:11:45.4106133Z 2022-03-17T20:11:45.4106252Z Generating XML reports... 2022-03-17T20:11:45.4150325Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201140.xml 2022-03-17T20:11:46.7150302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:46.7165630Z 2022-03-17T20:11:46.7165941Z Running tests... 2022-03-17T20:11:46.7166553Z ---------------------------------------------------------------------- 2022-03-17T20:11:46.7205460Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip (0.004s) 2022-03-17T20:11:46.7205991Z 2022-03-17T20:11:46.7206275Z ---------------------------------------------------------------------- 2022-03-17T20:11:46.7206612Z Ran 1 test in 0.004s 2022-03-17T20:11:46.7206775Z 2022-03-17T20:11:46.7206952Z OK (skipped=1) 2022-03-17T20:11:46.7207218Z 2022-03-17T20:11:46.7207376Z Generating XML reports... 2022-03-17T20:11:46.7250475Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201146.xml 2022-03-17T20:11:47.8130425Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:47.8147203Z 2022-03-17T20:11:47.8147656Z Running tests... 2022-03-17T20:11:47.8148130Z ---------------------------------------------------------------------- 2022-03-17T20:11:49.3609419Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:49.4032941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109069 2022-03-17T20:11:49.4156702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109070 2022-03-17T20:11:50.4986014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:50.4986810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:50.4987609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:50.4988234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:50.4994590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:50.4995501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:50.6201009Z ok (2.805s) 2022-03-17T20:11:50.6201217Z 2022-03-17T20:11:50.6201630Z ---------------------------------------------------------------------- 2022-03-17T20:11:50.6201945Z Ran 1 test in 2.805s 2022-03-17T20:11:50.6202108Z 2022-03-17T20:11:50.6202202Z OK 2022-03-17T20:11:50.6202338Z 2022-03-17T20:11:50.6202452Z Generating XML reports... 2022-03-17T20:11:50.6246784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201147.xml 2022-03-17T20:11:51.8911779Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:11:51.8926596Z 2022-03-17T20:11:51.8926856Z Running tests... 2022-03-17T20:11:51.8927359Z ---------------------------------------------------------------------- 2022-03-17T20:11:53.4310234Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:11:53.4731057Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109182 2022-03-17T20:11:53.4855938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109183 2022-03-17T20:11:54.5272522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:11:54.5280758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:11:54.5281796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:54.5373838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:11:54.5380350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:11:54.6295673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:11:57.1276499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.1277056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.1453407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.1453881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.1530484Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:11:57.1531540Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:11:57.1532557Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:11:57.1533329Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:11:57.1753120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.1753619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3106637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3107150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3280020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3280501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3505485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3505967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3679295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3679794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3953338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.3953817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.5214412Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:11:57.5214999Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:11:57.5215618Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:11:57.5216171Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:11:57.5358967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.5359419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.6759477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.6759972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.8159122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.8159608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.9510641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.9511147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.9682212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.9682711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.9954696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:57.9955169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.1305793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.1306419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.1477763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.1478254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.1703657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.1704262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.1875517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.1876168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.2147510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.2148030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.3552391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.3552909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.4949171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.4949691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.5349470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.5349963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:11:58.8973473Z ok (7.004s) 2022-03-17T20:11:58.8973657Z 2022-03-17T20:11:58.8974037Z ---------------------------------------------------------------------- 2022-03-17T20:11:58.8974373Z Ran 1 test in 7.005s 2022-03-17T20:11:58.8974539Z 2022-03-17T20:11:58.8974644Z OK 2022-03-17T20:11:58.8974762Z 2022-03-17T20:11:58.8974889Z Generating XML reports... 2022-03-17T20:11:58.9019857Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201151.xml 2022-03-17T20:12:00.1901637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:00.1917071Z 2022-03-17T20:12:00.1917565Z Running tests... 2022-03-17T20:12:00.1918051Z ---------------------------------------------------------------------- 2022-03-17T20:12:00.1935851Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:00.1936112Z 2022-03-17T20:12:00.1936423Z ---------------------------------------------------------------------- 2022-03-17T20:12:00.1936733Z Ran 1 test in 0.002s 2022-03-17T20:12:00.1936898Z 2022-03-17T20:12:00.1937019Z OK (skipped=1) 2022-03-17T20:12:00.1937146Z 2022-03-17T20:12:00.1937237Z Generating XML reports... 2022-03-17T20:12:00.1981996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201200.xml 2022-03-17T20:12:01.3438303Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:01.3453512Z 2022-03-17T20:12:01.3453818Z Running tests... 2022-03-17T20:12:01.3454425Z ---------------------------------------------------------------------- 2022-03-17T20:12:01.3473992Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:12:01.3474277Z 2022-03-17T20:12:01.3474627Z ---------------------------------------------------------------------- 2022-03-17T20:12:01.3475176Z Ran 1 test in 0.002s 2022-03-17T20:12:01.3475258Z 2022-03-17T20:12:01.3475377Z OK (skipped=1) 2022-03-17T20:12:01.3475532Z 2022-03-17T20:12:01.3475661Z Generating XML reports... 2022-03-17T20:12:01.3519891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201201.xml 2022-03-17T20:12:02.5007717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:02.5022994Z 2022-03-17T20:12:02.5023268Z Running tests... 2022-03-17T20:12:02.5023796Z ---------------------------------------------------------------------- 2022-03-17T20:12:04.0601611Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:12:04.1023486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109379 2022-03-17T20:12:04.1150698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109380 2022-03-17T20:12:05.1758104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:05.1812834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:05.1813957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:05.1859176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:05.1866471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:05.2828847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:08.2414535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.2415102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.2613878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.2618862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.2701497Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:08.2702327Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:08.2709084Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:08.2709878Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:08.2904071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.2906835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.3103337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.3109786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.4370402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.4374204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.4570116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:08.4577737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:09.0259467Z ok (6.523s) 2022-03-17T20:12:09.0259727Z 2022-03-17T20:12:09.0260072Z ---------------------------------------------------------------------- 2022-03-17T20:12:09.0260672Z Ran 1 test in 6.524s 2022-03-17T20:12:09.0261213Z 2022-03-17T20:12:09.0261282Z OK 2022-03-17T20:12:09.0261341Z 2022-03-17T20:12:09.0261480Z Generating XML reports... 2022-03-17T20:12:09.0306490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201202.xml 2022-03-17T20:12:10.3309330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:10.3325780Z 2022-03-17T20:12:10.3325923Z Running tests... 2022-03-17T20:12:10.3326418Z ---------------------------------------------------------------------- 2022-03-17T20:12:11.8594229Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:12:11.9018909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109506 2022-03-17T20:12:11.9142718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109507 2022-03-17T20:12:12.9670965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:13.0143768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:13.0144615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:13.0177510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:13.0185652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:13.1159783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:15.4349453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:15.4350407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:15.4484676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:15.4487204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:15.4547165Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:15.4548632Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:15.4550593Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:15.4552165Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:15.8237331Z ok (5.491s) 2022-03-17T20:12:15.8237576Z 2022-03-17T20:12:15.8237977Z ---------------------------------------------------------------------- 2022-03-17T20:12:15.8238289Z Ran 1 test in 5.491s 2022-03-17T20:12:15.8238459Z 2022-03-17T20:12:15.8238559Z OK 2022-03-17T20:12:15.8238701Z 2022-03-17T20:12:15.8238820Z Generating XML reports... 2022-03-17T20:12:15.8283573Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201210.xml 2022-03-17T20:12:17.1172845Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:17.1188125Z 2022-03-17T20:12:17.1188575Z Running tests... 2022-03-17T20:12:17.1189058Z ---------------------------------------------------------------------- 2022-03-17T20:12:18.6462756Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:12:18.6882583Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109633 2022-03-17T20:12:18.7007846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109634 2022-03-17T20:12:19.7406779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:19.7756233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:19.7757055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:19.7811908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:19.7818448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:19.8771916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:22.1971604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:22.1972141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:22.2122907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:22.2129888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:22.2186502Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:22.2187308Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:22.2203331Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:22.2204075Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:22.6114422Z ok (5.492s) 2022-03-17T20:12:22.6114661Z 2022-03-17T20:12:22.6115047Z ---------------------------------------------------------------------- 2022-03-17T20:12:22.6115366Z Ran 1 test in 5.493s 2022-03-17T20:12:22.6115531Z 2022-03-17T20:12:22.6115640Z OK 2022-03-17T20:12:22.6115775Z 2022-03-17T20:12:22.6115892Z Generating XML reports... 2022-03-17T20:12:22.6160065Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201217.xml 2022-03-17T20:12:23.9134751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:23.9150308Z 2022-03-17T20:12:23.9150625Z Running tests... 2022-03-17T20:12:23.9151069Z ---------------------------------------------------------------------- 2022-03-17T20:12:25.4709628Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:12:25.5129984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109760 2022-03-17T20:12:25.5257095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109761 2022-03-17T20:12:26.5883598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:26.6050683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:26.6051782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:26.6086852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:26.6093533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:26.7066106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:29.0378987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:29.0379517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:29.4347879Z ok (5.519s) 2022-03-17T20:12:29.4348485Z 2022-03-17T20:12:29.4348923Z ---------------------------------------------------------------------- 2022-03-17T20:12:29.4349249Z Ran 1 test in 5.520s 2022-03-17T20:12:29.4349414Z 2022-03-17T20:12:29.4349505Z OK 2022-03-17T20:12:29.4349638Z 2022-03-17T20:12:29.4349766Z Generating XML reports... 2022-03-17T20:12:29.4393493Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201223.xml 2022-03-17T20:12:30.7341332Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:30.7356516Z 2022-03-17T20:12:30.7356717Z Running tests... 2022-03-17T20:12:30.7357385Z ---------------------------------------------------------------------- 2022-03-17T20:12:32.2586219Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:12:32.3011407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109887 2022-03-17T20:12:32.3135663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109888 2022-03-17T20:12:33.3549303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:33.3723992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:33.3724947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:33.3751856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:33.3759002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:33.4738670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:36.3639032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:36.3651427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:36.3850191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:36.3860415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:36.3938984Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:36.3939746Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:36.3951423Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:36.3952141Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:36.9242645Z ok (6.188s) 2022-03-17T20:12:36.9243003Z 2022-03-17T20:12:36.9243640Z ---------------------------------------------------------------------- 2022-03-17T20:12:36.9244229Z Ran 1 test in 6.189s 2022-03-17T20:12:36.9244516Z 2022-03-17T20:12:36.9244678Z OK 2022-03-17T20:12:36.9244902Z 2022-03-17T20:12:36.9245130Z Generating XML reports... 2022-03-17T20:12:36.9290167Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201230.xml 2022-03-17T20:12:38.2239859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:38.2255599Z 2022-03-17T20:12:38.2256311Z Running tests... 2022-03-17T20:12:38.2256844Z ---------------------------------------------------------------------- 2022-03-17T20:12:39.7624225Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:12:39.8043849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110014 2022-03-17T20:12:39.8169269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110015 2022-03-17T20:12:40.8618418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:40.8618919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:40.8619702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:40.8620632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:40.8627750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:40.8628230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:43.5170639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:43.5171197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:43.5332855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:43.5333360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:43.5403614Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:43.5404402Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:43.5405445Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:43.5406222Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:43.9265913Z ok (5.701s) 2022-03-17T20:12:43.9266119Z 2022-03-17T20:12:43.9266509Z ---------------------------------------------------------------------- 2022-03-17T20:12:43.9266832Z Ran 1 test in 5.701s 2022-03-17T20:12:43.9267001Z 2022-03-17T20:12:43.9267097Z OK 2022-03-17T20:12:43.9267236Z 2022-03-17T20:12:43.9267358Z Generating XML reports... 2022-03-17T20:12:43.9313548Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201238.xml 2022-03-17T20:12:45.2291875Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:45.2308398Z 2022-03-17T20:12:45.2308581Z Running tests... 2022-03-17T20:12:45.2309026Z ---------------------------------------------------------------------- 2022-03-17T20:12:46.7589590Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:12:46.8013500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110141 2022-03-17T20:12:46.8137736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110142 2022-03-17T20:12:47.9117382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:47.9157193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:47.9157956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:47.9219008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:47.9226160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:48.0172186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:50.3292453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:50.3292969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:50.3428980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:50.3429469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:50.3490486Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:50.3491304Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:50.3492312Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:12:50.3493097Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:12:50.6230341Z ok (5.392s) 2022-03-17T20:12:50.6230777Z 2022-03-17T20:12:50.6231382Z ---------------------------------------------------------------------- 2022-03-17T20:12:50.6231961Z Ran 1 test in 5.392s 2022-03-17T20:12:50.6232295Z 2022-03-17T20:12:50.6232446Z OK 2022-03-17T20:12:50.6232651Z 2022-03-17T20:12:50.6232904Z Generating XML reports... 2022-03-17T20:12:50.6278313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201245.xml 2022-03-17T20:12:51.9164189Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:51.9181221Z 2022-03-17T20:12:51.9181433Z Running tests... 2022-03-17T20:12:51.9181895Z ---------------------------------------------------------------------- 2022-03-17T20:12:53.4406214Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:12:53.4825145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110268 2022-03-17T20:12:53.4948288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110269 2022-03-17T20:12:54.5603288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:12:54.5844313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:12:54.5845129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:54.5907327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:12:54.5914134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:12:54.6858656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:12:56.2895757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:56.2896291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:12:58.2051287Z ok (6.287s) 2022-03-17T20:12:58.2051693Z 2022-03-17T20:12:58.2052327Z ---------------------------------------------------------------------- 2022-03-17T20:12:58.2052875Z Ran 1 test in 6.287s 2022-03-17T20:12:58.2053229Z 2022-03-17T20:12:58.2053375Z OK 2022-03-17T20:12:58.2053612Z 2022-03-17T20:12:58.2053834Z Generating XML reports... 2022-03-17T20:12:58.2098647Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201251.xml 2022-03-17T20:12:59.5163386Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:12:59.5178476Z 2022-03-17T20:12:59.5178745Z Running tests... 2022-03-17T20:12:59.5179184Z ---------------------------------------------------------------------- 2022-03-17T20:13:01.0686539Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:13:01.1107332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110394 2022-03-17T20:13:01.1231841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110395 2022-03-17T20:13:02.1774220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:02.1781401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:02.1782191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:02.1875698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:02.1882430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:02.2793221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:02.5283746Z ok (3.010s) 2022-03-17T20:13:02.5284039Z 2022-03-17T20:13:02.5284354Z ---------------------------------------------------------------------- 2022-03-17T20:13:02.5284764Z Ran 1 test in 3.010s 2022-03-17T20:13:02.5284942Z 2022-03-17T20:13:02.5285039Z OK 2022-03-17T20:13:02.5285152Z 2022-03-17T20:13:02.5285298Z Generating XML reports... 2022-03-17T20:13:02.5329960Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201259.xml 2022-03-17T20:13:03.8052955Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:03.8068319Z 2022-03-17T20:13:03.8068573Z Running tests... 2022-03-17T20:13:03.8068998Z ---------------------------------------------------------------------- 2022-03-17T20:13:05.3387071Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:13:05.3806767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110507 2022-03-17T20:13:05.3931014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110508 2022-03-17T20:13:06.4385299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:06.4422203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:06.4423317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:06.4487017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:06.4493860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:06.5438082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:08.1601707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:13:08.1602243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:13:08.2152857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:13:08.2153312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:13:08.6012296Z ok (4.794s) 2022-03-17T20:13:08.6012512Z 2022-03-17T20:13:08.6012909Z ---------------------------------------------------------------------- 2022-03-17T20:13:08.6013263Z Ran 1 test in 4.794s 2022-03-17T20:13:08.6013440Z 2022-03-17T20:13:08.6013534Z OK 2022-03-17T20:13:08.6013668Z 2022-03-17T20:13:08.6013780Z Generating XML reports... 2022-03-17T20:13:08.6058423Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201303.xml 2022-03-17T20:13:09.9046464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:09.9061411Z 2022-03-17T20:13:09.9061935Z Running tests... 2022-03-17T20:13:09.9062524Z ---------------------------------------------------------------------- 2022-03-17T20:13:11.4589317Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:13:11.5011817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110633 2022-03-17T20:13:11.5137404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110634 2022-03-17T20:13:12.5556552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:12.5563657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:12.5565243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:12.5660664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:12.5667495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:12.6578815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:15.3229505Z ok (5.416s) 2022-03-17T20:13:15.3229882Z 2022-03-17T20:13:15.3230401Z ---------------------------------------------------------------------- 2022-03-17T20:13:15.3230705Z Ran 1 test in 5.417s 2022-03-17T20:13:15.3230838Z 2022-03-17T20:13:15.3230950Z OK 2022-03-17T20:13:15.3231575Z 2022-03-17T20:13:15.3231855Z Generating XML reports... 2022-03-17T20:13:15.3275300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201309.xml 2022-03-17T20:13:16.6261577Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:16.6277518Z 2022-03-17T20:13:16.6277982Z Running tests... 2022-03-17T20:13:16.6278491Z ---------------------------------------------------------------------- 2022-03-17T20:13:16.6300068Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:16.6300363Z 2022-03-17T20:13:16.6301010Z ---------------------------------------------------------------------- 2022-03-17T20:13:16.6301354Z Ran 1 test in 0.002s 2022-03-17T20:13:16.6301522Z 2022-03-17T20:13:16.6301631Z OK (skipped=1) 2022-03-17T20:13:16.6301769Z 2022-03-17T20:13:16.6301894Z Generating XML reports... 2022-03-17T20:13:16.6345596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201316.xml 2022-03-17T20:13:17.7747938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:17.7762670Z 2022-03-17T20:13:17.7762868Z Running tests... 2022-03-17T20:13:17.7763673Z ---------------------------------------------------------------------- 2022-03-17T20:13:17.7781696Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:13:19.3092362Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:13:19.3515017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110791 2022-03-17T20:13:19.3638648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110792 2022-03-17T20:13:20.4116328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:20.4423929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:20.4424747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:20.4521643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:20.4528264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:20.5439470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:23.4736516Z ok (5.697s) 2022-03-17T20:13:23.4736764Z 2022-03-17T20:13:23.4737129Z ---------------------------------------------------------------------- 2022-03-17T20:13:23.4737462Z Ran 1 test in 5.697s 2022-03-17T20:13:23.4737616Z 2022-03-17T20:13:23.4737733Z OK 2022-03-17T20:13:23.4737848Z 2022-03-17T20:13:23.4737987Z Generating XML reports... 2022-03-17T20:13:23.4782495Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201317.xml 2022-03-17T20:13:24.7736890Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:24.7753354Z 2022-03-17T20:13:24.7753713Z Running tests... 2022-03-17T20:13:24.7754177Z ---------------------------------------------------------------------- 2022-03-17T20:13:24.7777704Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:13:26.3121729Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:13:26.3561790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110918 2022-03-17T20:13:26.3691188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110919 2022-03-17T20:13:27.4189654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:27.4640939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:27.4641712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:27.4696165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:27.4703467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:27.5656190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:30.1883576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:13:30.1888103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:13:30.5807509Z ok (5.805s) 2022-03-17T20:13:30.5807702Z 2022-03-17T20:13:30.5808172Z ---------------------------------------------------------------------- 2022-03-17T20:13:30.5808482Z Ran 1 test in 5.805s 2022-03-17T20:13:30.5808647Z 2022-03-17T20:13:30.5808741Z OK 2022-03-17T20:13:30.5809031Z 2022-03-17T20:13:30.5809173Z Generating XML reports... 2022-03-17T20:13:30.5853953Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201324.xml 2022-03-17T20:13:31.8708588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:31.8723889Z 2022-03-17T20:13:31.8724321Z Running tests... 2022-03-17T20:13:31.8724820Z ---------------------------------------------------------------------- 2022-03-17T20:13:31.8750185Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:13:33.3932625Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:13:33.4349843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111045 2022-03-17T20:13:33.4472757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111046 2022-03-17T20:13:34.4828093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:34.5238642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:34.5239468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:34.5334580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:34.5341229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:34.6254216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:37.1981451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:13:37.1981999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:13:37.5569925Z ok (5.684s) 2022-03-17T20:13:37.5570247Z 2022-03-17T20:13:37.5570933Z ---------------------------------------------------------------------- 2022-03-17T20:13:37.5571509Z Ran 1 test in 5.685s 2022-03-17T20:13:37.5571804Z 2022-03-17T20:13:37.5571975Z OK 2022-03-17T20:13:37.5572225Z 2022-03-17T20:13:37.5572452Z Generating XML reports... 2022-03-17T20:13:37.5618621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201331.xml 2022-03-17T20:13:38.8559854Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:38.8574834Z 2022-03-17T20:13:38.8575115Z Running tests... 2022-03-17T20:13:38.8575579Z ---------------------------------------------------------------------- 2022-03-17T20:13:38.8594832Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:13:40.3840122Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:13:40.4266778Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111172 2022-03-17T20:13:40.4391695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111173 2022-03-17T20:13:41.5134608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:41.5465386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:41.5466849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:41.5539172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:41.5546292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:13:41.6482508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:44.5490031Z ok (5.691s) 2022-03-17T20:13:44.5490300Z 2022-03-17T20:13:44.5490689Z ---------------------------------------------------------------------- 2022-03-17T20:13:44.5491031Z Ran 1 test in 5.691s 2022-03-17T20:13:44.5491195Z 2022-03-17T20:13:44.5491298Z OK 2022-03-17T20:13:44.5491432Z 2022-03-17T20:13:44.5491563Z Generating XML reports... 2022-03-17T20:13:44.5536482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201338.xml 2022-03-17T20:13:45.8496582Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:45.8512200Z 2022-03-17T20:13:45.8512497Z Running tests... 2022-03-17T20:13:45.8512941Z ---------------------------------------------------------------------- 2022-03-17T20:13:45.8532446Z test_all_gather (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:45.8532704Z 2022-03-17T20:13:45.8533000Z ---------------------------------------------------------------------- 2022-03-17T20:13:45.8533315Z Ran 1 test in 0.002s 2022-03-17T20:13:45.8533484Z 2022-03-17T20:13:45.8533591Z OK (skipped=1) 2022-03-17T20:13:45.8533748Z 2022-03-17T20:13:45.8534764Z Generating XML reports... 2022-03-17T20:13:45.8578304Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201345.xml 2022-03-17T20:13:47.0059067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:47.0074301Z 2022-03-17T20:13:47.0074620Z Running tests... 2022-03-17T20:13:47.0075064Z ---------------------------------------------------------------------- 2022-03-17T20:13:47.0094888Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:47.0095176Z 2022-03-17T20:13:47.0095464Z ---------------------------------------------------------------------- 2022-03-17T20:13:47.0095808Z Ran 1 test in 0.002s 2022-03-17T20:13:47.0095953Z 2022-03-17T20:13:47.0096090Z OK (skipped=1) 2022-03-17T20:13:47.0096230Z 2022-03-17T20:13:47.0096350Z Generating XML reports... 2022-03-17T20:13:47.0140037Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201347.xml 2022-03-17T20:13:48.1606095Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:48.1621582Z 2022-03-17T20:13:48.1621729Z Running tests... 2022-03-17T20:13:48.1622171Z ---------------------------------------------------------------------- 2022-03-17T20:13:48.1642209Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:48.1642804Z 2022-03-17T20:13:48.1643287Z ---------------------------------------------------------------------- 2022-03-17T20:13:48.1643659Z Ran 1 test in 0.002s 2022-03-17T20:13:48.1643836Z 2022-03-17T20:13:48.1643938Z OK (skipped=1) 2022-03-17T20:13:48.1644082Z 2022-03-17T20:13:48.1644208Z Generating XML reports... 2022-03-17T20:13:48.1688433Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201348.xml 2022-03-17T20:13:49.3032769Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:49.3048074Z 2022-03-17T20:13:49.3048378Z Running tests... 2022-03-17T20:13:49.3048805Z ---------------------------------------------------------------------- 2022-03-17T20:13:49.3068624Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:49.3068907Z 2022-03-17T20:13:49.3069193Z ---------------------------------------------------------------------- 2022-03-17T20:13:49.3069504Z Ran 1 test in 0.002s 2022-03-17T20:13:49.3069664Z 2022-03-17T20:13:49.3069774Z OK (skipped=1) 2022-03-17T20:13:49.3069928Z 2022-03-17T20:13:49.3070052Z Generating XML reports... 2022-03-17T20:13:49.3112788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201349.xml 2022-03-17T20:13:50.4528733Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:50.4544102Z 2022-03-17T20:13:50.4544535Z Running tests... 2022-03-17T20:13:50.4545077Z ---------------------------------------------------------------------- 2022-03-17T20:13:50.4564414Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:50.4564673Z 2022-03-17T20:13:50.4564996Z ---------------------------------------------------------------------- 2022-03-17T20:13:50.4565317Z Ran 1 test in 0.002s 2022-03-17T20:13:50.4565472Z 2022-03-17T20:13:50.4565588Z OK (skipped=1) 2022-03-17T20:13:50.4565723Z 2022-03-17T20:13:50.4565844Z Generating XML reports... 2022-03-17T20:13:50.4608555Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201350.xml 2022-03-17T20:13:51.5947786Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:51.5963538Z 2022-03-17T20:13:51.5963695Z Running tests... 2022-03-17T20:13:51.5964410Z ---------------------------------------------------------------------- 2022-03-17T20:13:51.5996386Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:13:51.5996692Z 2022-03-17T20:13:51.5997023Z ---------------------------------------------------------------------- 2022-03-17T20:13:51.5997332Z Ran 1 test in 0.003s 2022-03-17T20:13:51.5997479Z 2022-03-17T20:13:51.5997585Z OK (skipped=1) 2022-03-17T20:13:51.5997715Z 2022-03-17T20:13:51.5997837Z Generating XML reports... 2022-03-17T20:13:51.6041670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201351.xml 2022-03-17T20:13:52.7472710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:52.7488406Z 2022-03-17T20:13:52.7488550Z Running tests... 2022-03-17T20:13:52.7489271Z ---------------------------------------------------------------------- 2022-03-17T20:13:52.7509164Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:13:52.7509654Z 2022-03-17T20:13:52.7510160Z ---------------------------------------------------------------------- 2022-03-17T20:13:52.7510504Z Ran 1 test in 0.002s 2022-03-17T20:13:52.7510671Z 2022-03-17T20:13:52.7510785Z OK (skipped=1) 2022-03-17T20:13:52.7510896Z 2022-03-17T20:13:52.7511020Z Generating XML reports... 2022-03-17T20:13:52.7555250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201352.xml 2022-03-17T20:13:53.8970637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:13:53.8986242Z 2022-03-17T20:13:53.8986371Z Running tests... 2022-03-17T20:13:53.8987073Z ---------------------------------------------------------------------- 2022-03-17T20:13:55.4447536Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:13:55.4868497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111544 2022-03-17T20:13:55.4994287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111545 2022-03-17T20:13:56.5512794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:13:56.5881093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:13:56.5881888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:56.5918138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:13:56.5925056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:13:56.6896618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:00.6106334Z ok (6.712s) 2022-03-17T20:14:00.6106745Z 2022-03-17T20:14:00.6107377Z ---------------------------------------------------------------------- 2022-03-17T20:14:00.6108027Z Ran 1 test in 6.712s 2022-03-17T20:14:00.6108299Z 2022-03-17T20:14:00.6108466Z OK 2022-03-17T20:14:00.6108728Z 2022-03-17T20:14:00.6108959Z Generating XML reports... 2022-03-17T20:14:00.6153675Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201353.xml 2022-03-17T20:14:01.9102698Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:01.9118449Z 2022-03-17T20:14:01.9118826Z Running tests... 2022-03-17T20:14:01.9119349Z ---------------------------------------------------------------------- 2022-03-17T20:14:03.4455822Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:14:03.4875431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111672 2022-03-17T20:14:03.5000829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111673 2022-03-17T20:14:04.5371902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:04.5448203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:04.5449422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:04.5473285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:04.5480287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:04.6463478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:08.6114179Z ok (6.699s) 2022-03-17T20:14:08.6114559Z 2022-03-17T20:14:08.6115104Z ---------------------------------------------------------------------- 2022-03-17T20:14:08.6115466Z Ran 1 test in 6.699s 2022-03-17T20:14:08.6115612Z 2022-03-17T20:14:08.6115707Z OK 2022-03-17T20:14:08.6115844Z 2022-03-17T20:14:08.6115980Z Generating XML reports... 2022-03-17T20:14:08.6160164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201401.xml 2022-03-17T20:14:09.9101960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:09.9118935Z 2022-03-17T20:14:09.9119452Z Running tests... 2022-03-17T20:14:09.9120058Z ---------------------------------------------------------------------- 2022-03-17T20:14:09.9139615Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:09.9140147Z 2022-03-17T20:14:09.9141077Z ---------------------------------------------------------------------- 2022-03-17T20:14:09.9141749Z Ran 1 test in 0.002s 2022-03-17T20:14:09.9142032Z 2022-03-17T20:14:09.9142220Z OK (skipped=1) 2022-03-17T20:14:09.9142933Z 2022-03-17T20:14:09.9143164Z Generating XML reports... 2022-03-17T20:14:09.9188527Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201409.xml 2022-03-17T20:14:11.0662208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:11.0678715Z 2022-03-17T20:14:11.0679162Z Running tests... 2022-03-17T20:14:11.0679658Z ---------------------------------------------------------------------- 2022-03-17T20:14:11.0699812Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:11.0700023Z 2022-03-17T20:14:11.0701418Z ---------------------------------------------------------------------- 2022-03-17T20:14:11.0701787Z Ran 1 test in 0.002s 2022-03-17T20:14:11.0701956Z 2022-03-17T20:14:11.0702091Z OK (skipped=1) 2022-03-17T20:14:11.0702220Z 2022-03-17T20:14:11.0702350Z Generating XML reports... 2022-03-17T20:14:11.0746116Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201411.xml 2022-03-17T20:14:12.1765659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:12.1781522Z 2022-03-17T20:14:12.1781727Z Running tests... 2022-03-17T20:14:12.1782421Z ---------------------------------------------------------------------- 2022-03-17T20:14:13.7247037Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:14:13.7672208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111870 2022-03-17T20:14:13.7796911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111871 2022-03-17T20:14:14.8220120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:14.8229129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:14.8230594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:14.8320886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:14.8327328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:14.9245364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:17.0879778Z ok (4.909s) 2022-03-17T20:14:17.0880045Z 2022-03-17T20:14:17.0880430Z ---------------------------------------------------------------------- 2022-03-17T20:14:17.0880782Z Ran 1 test in 4.910s 2022-03-17T20:14:17.0880937Z 2022-03-17T20:14:17.0881028Z OK 2022-03-17T20:14:17.0881135Z 2022-03-17T20:14:17.0881267Z Generating XML reports... 2022-03-17T20:14:17.0925046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201412.xml 2022-03-17T20:14:18.3935737Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:18.3951625Z 2022-03-17T20:14:18.3951841Z Running tests... 2022-03-17T20:14:18.3952278Z ---------------------------------------------------------------------- 2022-03-17T20:14:19.9390532Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:14:19.9814234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111996 2022-03-17T20:14:19.9938173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111997 2022-03-17T20:14:21.0350036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:21.0661926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:21.0663142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:21.0756565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:21.0763372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:21.1677033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:23.4025394Z ok (5.007s) 2022-03-17T20:14:23.4025720Z 2022-03-17T20:14:23.4026051Z ---------------------------------------------------------------------- 2022-03-17T20:14:23.4026351Z Ran 1 test in 5.007s 2022-03-17T20:14:23.4026517Z 2022-03-17T20:14:23.4026872Z OK 2022-03-17T20:14:23.4027078Z 2022-03-17T20:14:23.4027146Z Generating XML reports... 2022-03-17T20:14:23.4070638Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201418.xml 2022-03-17T20:14:24.7057579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:24.7073031Z 2022-03-17T20:14:24.7073439Z Running tests... 2022-03-17T20:14:24.7073928Z ---------------------------------------------------------------------- 2022-03-17T20:14:26.2453121Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:14:26.2877064Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112122 2022-03-17T20:14:26.3002229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112123 2022-03-17T20:14:27.3385253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:27.3743218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:27.3744029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:27.3791015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:27.3797557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:27.4758235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:30.1093926Z ok (5.402s) 2022-03-17T20:14:30.1094145Z 2022-03-17T20:14:30.1094544Z ---------------------------------------------------------------------- 2022-03-17T20:14:30.1094864Z Ran 1 test in 5.402s 2022-03-17T20:14:30.1095028Z 2022-03-17T20:14:30.1095127Z OK 2022-03-17T20:14:30.1095263Z 2022-03-17T20:14:30.1095422Z Generating XML reports... 2022-03-17T20:14:30.1141317Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201424.xml 2022-03-17T20:14:31.4133179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:31.4148665Z 2022-03-17T20:14:31.4149287Z Running tests... 2022-03-17T20:14:31.4149769Z ---------------------------------------------------------------------- 2022-03-17T20:14:32.9666574Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:14:33.0090267Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112245 2022-03-17T20:14:33.0216173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112246 2022-03-17T20:14:34.0639115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:14:34.0872661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:14:34.0873467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:34.0944629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:14:34.0951771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:14:34.1884461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:14:34.2073500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:14:34.2074039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:14:34.2074665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:14:34.2075521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:14:37.5899302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:14:37.5899884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:14:37.5900852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:14:37.5901549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:14:37.6350807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:14:37.6351310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:14:37.6352060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:14:37.6352742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:14:38.1332459Z ok (6.718s) 2022-03-17T20:14:38.1332865Z 2022-03-17T20:14:38.1333314Z ---------------------------------------------------------------------- 2022-03-17T20:14:38.1333659Z Ran 1 test in 6.718s 2022-03-17T20:14:38.1333806Z 2022-03-17T20:14:38.1333912Z OK 2022-03-17T20:14:38.1334048Z 2022-03-17T20:14:38.1334183Z Generating XML reports... 2022-03-17T20:14:38.1381082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201431.xml 2022-03-17T20:14:39.4247057Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:39.4261926Z 2022-03-17T20:14:39.4262133Z Running tests... 2022-03-17T20:14:39.4262642Z ---------------------------------------------------------------------- 2022-03-17T20:14:39.4282816Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:39.4283098Z 2022-03-17T20:14:39.4283411Z ---------------------------------------------------------------------- 2022-03-17T20:14:39.4283742Z Ran 1 test in 0.002s 2022-03-17T20:14:39.4283902Z 2022-03-17T20:14:39.4283998Z OK (skipped=1) 2022-03-17T20:14:39.4284097Z 2022-03-17T20:14:39.4284176Z Generating XML reports... 2022-03-17T20:14:39.4326550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201439.xml 2022-03-17T20:14:40.5673979Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:40.5689511Z 2022-03-17T20:14:40.5689887Z Running tests... 2022-03-17T20:14:40.5690383Z ---------------------------------------------------------------------- 2022-03-17T20:14:40.5711884Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:40.5712200Z 2022-03-17T20:14:40.5712525Z ---------------------------------------------------------------------- 2022-03-17T20:14:40.5712835Z Ran 1 test in 0.002s 2022-03-17T20:14:40.5713276Z 2022-03-17T20:14:40.5713382Z OK (skipped=1) 2022-03-17T20:14:40.5713580Z 2022-03-17T20:14:40.5713709Z Generating XML reports... 2022-03-17T20:14:40.5759241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201440.xml 2022-03-17T20:14:41.7208751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:41.7223485Z 2022-03-17T20:14:41.7223783Z Running tests... 2022-03-17T20:14:41.7224744Z ---------------------------------------------------------------------- 2022-03-17T20:14:41.7244758Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:41.7245857Z 2022-03-17T20:14:41.7246207Z ---------------------------------------------------------------------- 2022-03-17T20:14:41.7246563Z Ran 1 test in 0.002s 2022-03-17T20:14:41.7246708Z 2022-03-17T20:14:41.7246818Z OK (skipped=1) 2022-03-17T20:14:41.7246955Z 2022-03-17T20:14:41.7247089Z Generating XML reports... 2022-03-17T20:14:41.7289809Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201441.xml 2022-03-17T20:14:42.8693028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:42.8707941Z 2022-03-17T20:14:42.8708224Z Running tests... 2022-03-17T20:14:42.8708662Z ---------------------------------------------------------------------- 2022-03-17T20:14:42.8728491Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:42.8728822Z 2022-03-17T20:14:42.8729117Z ---------------------------------------------------------------------- 2022-03-17T20:14:42.8729454Z Ran 1 test in 0.002s 2022-03-17T20:14:42.8729618Z 2022-03-17T20:14:42.8729716Z OK (skipped=1) 2022-03-17T20:14:42.8729866Z 2022-03-17T20:14:42.8729986Z Generating XML reports... 2022-03-17T20:14:42.8773533Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201442.xml 2022-03-17T20:14:43.9673100Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:43.9688112Z 2022-03-17T20:14:43.9688479Z Running tests... 2022-03-17T20:14:43.9688972Z ---------------------------------------------------------------------- 2022-03-17T20:14:43.9709369Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:43.9709625Z 2022-03-17T20:14:43.9709872Z ---------------------------------------------------------------------- 2022-03-17T20:14:43.9710214Z Ran 1 test in 0.002s 2022-03-17T20:14:43.9710410Z 2022-03-17T20:14:43.9710508Z OK (skipped=1) 2022-03-17T20:14:43.9710671Z 2022-03-17T20:14:43.9710790Z Generating XML reports... 2022-03-17T20:14:43.9754098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201443.xml 2022-03-17T20:14:45.1077554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:45.1092305Z 2022-03-17T20:14:45.1092555Z Running tests... 2022-03-17T20:14:45.1093011Z ---------------------------------------------------------------------- 2022-03-17T20:14:45.1113599Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:45.1114151Z 2022-03-17T20:14:45.1114480Z ---------------------------------------------------------------------- 2022-03-17T20:14:45.1114820Z Ran 1 test in 0.002s 2022-03-17T20:14:45.1114880Z 2022-03-17T20:14:45.1114992Z OK (skipped=1) 2022-03-17T20:14:45.1115128Z 2022-03-17T20:14:45.1115251Z Generating XML reports... 2022-03-17T20:14:45.1158058Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201445.xml 2022-03-17T20:14:46.2555257Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:46.2570094Z 2022-03-17T20:14:46.2570437Z Running tests... 2022-03-17T20:14:46.2570902Z ---------------------------------------------------------------------- 2022-03-17T20:14:46.2591833Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:46.2592082Z 2022-03-17T20:14:46.2592387Z ---------------------------------------------------------------------- 2022-03-17T20:14:46.2592700Z Ran 1 test in 0.002s 2022-03-17T20:14:46.2592843Z 2022-03-17T20:14:46.2592954Z OK (skipped=1) 2022-03-17T20:14:46.2593105Z 2022-03-17T20:14:46.2593230Z Generating XML reports... 2022-03-17T20:14:46.2636614Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201446.xml 2022-03-17T20:14:47.4056235Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:47.4072017Z 2022-03-17T20:14:47.4072508Z Running tests... 2022-03-17T20:14:47.4072996Z ---------------------------------------------------------------------- 2022-03-17T20:14:47.4092942Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:47.4093233Z 2022-03-17T20:14:47.4093702Z ---------------------------------------------------------------------- 2022-03-17T20:14:47.4094094Z Ran 1 test in 0.002s 2022-03-17T20:14:47.4094261Z 2022-03-17T20:14:47.4094368Z OK (skipped=1) 2022-03-17T20:14:47.4094523Z 2022-03-17T20:14:47.4094647Z Generating XML reports... 2022-03-17T20:14:47.4137965Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201447.xml 2022-03-17T20:14:48.5125338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:48.5140557Z 2022-03-17T20:14:48.5140788Z Running tests... 2022-03-17T20:14:48.5141700Z ---------------------------------------------------------------------- 2022-03-17T20:14:48.5162246Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:48.5162870Z 2022-03-17T20:14:48.5163491Z ---------------------------------------------------------------------- 2022-03-17T20:14:48.5163957Z Ran 1 test in 0.002s 2022-03-17T20:14:48.5164119Z 2022-03-17T20:14:48.5164209Z OK (skipped=1) 2022-03-17T20:14:48.5164360Z 2022-03-17T20:14:48.5164490Z Generating XML reports... 2022-03-17T20:14:48.5209148Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201448.xml 2022-03-17T20:14:49.6674555Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:49.6689466Z 2022-03-17T20:14:49.6689644Z Running tests... 2022-03-17T20:14:49.6690238Z ---------------------------------------------------------------------- 2022-03-17T20:14:49.6711595Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:49.6711901Z 2022-03-17T20:14:49.6712184Z ---------------------------------------------------------------------- 2022-03-17T20:14:49.6712702Z Ran 1 test in 0.002s 2022-03-17T20:14:49.6712837Z 2022-03-17T20:14:49.6712945Z OK (skipped=1) 2022-03-17T20:14:49.6713082Z 2022-03-17T20:14:49.6713203Z Generating XML reports... 2022-03-17T20:14:49.6756121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201449.xml 2022-03-17T20:14:50.8232998Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:50.8247600Z 2022-03-17T20:14:50.8248126Z Running tests... 2022-03-17T20:14:50.8248652Z ---------------------------------------------------------------------- 2022-03-17T20:14:50.8269425Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:50.8269749Z 2022-03-17T20:14:50.8270047Z ---------------------------------------------------------------------- 2022-03-17T20:14:50.8270385Z Ran 1 test in 0.002s 2022-03-17T20:14:50.8270781Z 2022-03-17T20:14:50.8270901Z OK (skipped=1) 2022-03-17T20:14:50.8271030Z 2022-03-17T20:14:50.8271155Z Generating XML reports... 2022-03-17T20:14:50.8314548Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201450.xml 2022-03-17T20:14:51.9781882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:51.9797233Z 2022-03-17T20:14:51.9797507Z Running tests... 2022-03-17T20:14:51.9798145Z ---------------------------------------------------------------------- 2022-03-17T20:14:51.9818534Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:51.9819468Z 2022-03-17T20:14:51.9820107Z ---------------------------------------------------------------------- 2022-03-17T20:14:51.9821107Z Ran 1 test in 0.002s 2022-03-17T20:14:51.9821320Z 2022-03-17T20:14:51.9821442Z OK (skipped=1) 2022-03-17T20:14:51.9821595Z 2022-03-17T20:14:51.9821729Z Generating XML reports... 2022-03-17T20:14:51.9864254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201451.xml 2022-03-17T20:14:53.1312624Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:53.1327736Z 2022-03-17T20:14:53.1327892Z Running tests... 2022-03-17T20:14:53.1328857Z ---------------------------------------------------------------------- 2022-03-17T20:14:53.1349917Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:53.1350500Z 2022-03-17T20:14:53.1351113Z ---------------------------------------------------------------------- 2022-03-17T20:14:53.1351530Z Ran 1 test in 0.002s 2022-03-17T20:14:53.1351693Z 2022-03-17T20:14:53.1351799Z OK (skipped=1) 2022-03-17T20:14:53.1351953Z 2022-03-17T20:14:53.1352076Z Generating XML reports... 2022-03-17T20:14:53.1395608Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201453.xml 2022-03-17T20:14:54.2875966Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:54.2891258Z 2022-03-17T20:14:54.2891385Z Running tests... 2022-03-17T20:14:54.2892080Z ---------------------------------------------------------------------- 2022-03-17T20:14:54.2915955Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:54.2916280Z 2022-03-17T20:14:54.2916542Z ---------------------------------------------------------------------- 2022-03-17T20:14:54.2916889Z Ran 1 test in 0.003s 2022-03-17T20:14:54.2917055Z 2022-03-17T20:14:54.2917179Z OK (skipped=1) 2022-03-17T20:14:54.2917339Z 2022-03-17T20:14:54.2917432Z Generating XML reports... 2022-03-17T20:14:54.2961608Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201454.xml 2022-03-17T20:14:55.3875442Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:55.3890587Z 2022-03-17T20:14:55.3890808Z Running tests... 2022-03-17T20:14:55.3891245Z ---------------------------------------------------------------------- 2022-03-17T20:14:55.3912074Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:55.3912357Z 2022-03-17T20:14:55.3912639Z ---------------------------------------------------------------------- 2022-03-17T20:14:55.3912970Z Ran 1 test in 0.002s 2022-03-17T20:14:55.3913114Z 2022-03-17T20:14:55.3913224Z OK (skipped=1) 2022-03-17T20:14:55.3913379Z 2022-03-17T20:14:55.3913503Z Generating XML reports... 2022-03-17T20:14:55.3957690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201455.xml 2022-03-17T20:14:56.5336773Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:56.5352803Z 2022-03-17T20:14:56.5353127Z Running tests... 2022-03-17T20:14:56.5353583Z ---------------------------------------------------------------------- 2022-03-17T20:14:56.5373856Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:56.5374112Z 2022-03-17T20:14:56.5374443Z ---------------------------------------------------------------------- 2022-03-17T20:14:56.5374766Z Ran 1 test in 0.002s 2022-03-17T20:14:56.5374911Z 2022-03-17T20:14:56.5375022Z OK (skipped=1) 2022-03-17T20:14:56.5375130Z 2022-03-17T20:14:56.5375260Z Generating XML reports... 2022-03-17T20:14:56.5418909Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201456.xml 2022-03-17T20:14:57.6719746Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:57.6735497Z 2022-03-17T20:14:57.6735775Z Running tests... 2022-03-17T20:14:57.6736486Z ---------------------------------------------------------------------- 2022-03-17T20:14:57.6758543Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:57.6758909Z 2022-03-17T20:14:57.6759191Z ---------------------------------------------------------------------- 2022-03-17T20:14:57.6759553Z Ran 1 test in 0.002s 2022-03-17T20:14:57.6759786Z 2022-03-17T20:14:57.6759940Z OK (skipped=1) 2022-03-17T20:14:57.6760101Z 2022-03-17T20:14:57.6760224Z Generating XML reports... 2022-03-17T20:14:57.6803275Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201457.xml 2022-03-17T20:14:58.8196781Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:58.8212464Z 2022-03-17T20:14:58.8212655Z Running tests... 2022-03-17T20:14:58.8213117Z ---------------------------------------------------------------------- 2022-03-17T20:14:58.8234931Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:58.8235226Z 2022-03-17T20:14:58.8235518Z ---------------------------------------------------------------------- 2022-03-17T20:14:58.8235844Z Ran 1 test in 0.002s 2022-03-17T20:14:58.8236010Z 2022-03-17T20:14:58.8236097Z OK (skipped=1) 2022-03-17T20:14:58.8236248Z 2022-03-17T20:14:58.8236368Z Generating XML reports... 2022-03-17T20:14:58.8279996Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201458.xml 2022-03-17T20:14:59.9707052Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:14:59.9722857Z 2022-03-17T20:14:59.9723306Z Running tests... 2022-03-17T20:14:59.9723797Z ---------------------------------------------------------------------- 2022-03-17T20:14:59.9744585Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:14:59.9744867Z 2022-03-17T20:14:59.9745149Z ---------------------------------------------------------------------- 2022-03-17T20:14:59.9745474Z Ran 1 test in 0.002s 2022-03-17T20:14:59.9745635Z 2022-03-17T20:14:59.9745744Z OK (skipped=1) 2022-03-17T20:14:59.9745900Z 2022-03-17T20:14:59.9746007Z Generating XML reports... 2022-03-17T20:14:59.9790430Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201459.xml 2022-03-17T20:15:01.1178184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:01.1194398Z 2022-03-17T20:15:01.1194875Z Running tests... 2022-03-17T20:15:01.1195542Z ---------------------------------------------------------------------- 2022-03-17T20:15:01.1216519Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:01.1216806Z 2022-03-17T20:15:01.1217122Z ---------------------------------------------------------------------- 2022-03-17T20:15:01.1217442Z Ran 1 test in 0.002s 2022-03-17T20:15:01.1217946Z 2022-03-17T20:15:01.1218064Z OK (skipped=1) 2022-03-17T20:15:01.1218219Z 2022-03-17T20:15:01.1218343Z Generating XML reports... 2022-03-17T20:15:01.1262853Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201501.xml 2022-03-17T20:15:02.2901960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:02.2918355Z 2022-03-17T20:15:02.2919028Z Running tests... 2022-03-17T20:15:02.2919520Z ---------------------------------------------------------------------- 2022-03-17T20:15:02.2941442Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:02.2941708Z 2022-03-17T20:15:02.2942298Z ---------------------------------------------------------------------- 2022-03-17T20:15:02.2942627Z Ran 1 test in 0.002s 2022-03-17T20:15:02.2942797Z 2022-03-17T20:15:02.2942916Z OK (skipped=1) 2022-03-17T20:15:02.2943070Z 2022-03-17T20:15:02.2943197Z Generating XML reports... 2022-03-17T20:15:02.2987406Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201502.xml 2022-03-17T20:15:03.4463841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:03.4478951Z 2022-03-17T20:15:03.4479267Z Running tests... 2022-03-17T20:15:03.4479695Z ---------------------------------------------------------------------- 2022-03-17T20:15:03.4501351Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:03.4501766Z 2022-03-17T20:15:03.4502114Z ---------------------------------------------------------------------- 2022-03-17T20:15:03.4502376Z Ran 1 test in 0.002s 2022-03-17T20:15:03.4502534Z 2022-03-17T20:15:03.4502660Z OK (skipped=1) 2022-03-17T20:15:03.4502845Z 2022-03-17T20:15:03.4502920Z Generating XML reports... 2022-03-17T20:15:03.4546618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201503.xml 2022-03-17T20:15:04.5958118Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:04.5972913Z 2022-03-17T20:15:04.5973069Z Running tests... 2022-03-17T20:15:04.5974370Z ---------------------------------------------------------------------- 2022-03-17T20:15:04.5994809Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:04.5995354Z 2022-03-17T20:15:04.5995941Z ---------------------------------------------------------------------- 2022-03-17T20:15:04.5996467Z Ran 1 test in 0.002s 2022-03-17T20:15:04.5996632Z 2022-03-17T20:15:04.5996744Z OK (skipped=1) 2022-03-17T20:15:04.5996900Z 2022-03-17T20:15:04.5997030Z Generating XML reports... 2022-03-17T20:15:04.6041090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201504.xml 2022-03-17T20:15:05.7341875Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:05.7356515Z 2022-03-17T20:15:05.7356671Z Running tests... 2022-03-17T20:15:05.7357545Z ---------------------------------------------------------------------- 2022-03-17T20:15:05.7377227Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:05.7377624Z 2022-03-17T20:15:05.7378036Z ---------------------------------------------------------------------- 2022-03-17T20:15:05.7378348Z Ran 1 test in 0.002s 2022-03-17T20:15:05.7378511Z 2022-03-17T20:15:05.7378619Z OK (skipped=1) 2022-03-17T20:15:05.7378776Z 2022-03-17T20:15:05.7378884Z Generating XML reports... 2022-03-17T20:15:05.7421589Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201505.xml 2022-03-17T20:15:06.8809788Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:06.8824486Z 2022-03-17T20:15:06.8824882Z Running tests... 2022-03-17T20:15:06.8826143Z ---------------------------------------------------------------------- 2022-03-17T20:15:06.8847888Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:06.8848501Z 2022-03-17T20:15:06.8849081Z ---------------------------------------------------------------------- 2022-03-17T20:15:06.8849580Z Ran 1 test in 0.002s 2022-03-17T20:15:06.8849747Z 2022-03-17T20:15:06.8849856Z OK (skipped=1) 2022-03-17T20:15:06.8849992Z 2022-03-17T20:15:06.8850117Z Generating XML reports... 2022-03-17T20:15:06.8893253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201506.xml 2022-03-17T20:15:08.0236498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:08.0250957Z 2022-03-17T20:15:08.0251222Z Running tests... 2022-03-17T20:15:08.0251668Z ---------------------------------------------------------------------- 2022-03-17T20:15:08.0275824Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:08.0276092Z 2022-03-17T20:15:08.0276533Z ---------------------------------------------------------------------- 2022-03-17T20:15:08.0277111Z Ran 1 test in 0.003s 2022-03-17T20:15:08.0277291Z 2022-03-17T20:15:08.0277392Z OK (skipped=1) 2022-03-17T20:15:08.0277518Z 2022-03-17T20:15:08.0277643Z Generating XML reports... 2022-03-17T20:15:08.0320768Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201508.xml 2022-03-17T20:15:09.1808731Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:09.1824253Z 2022-03-17T20:15:09.1824701Z Running tests... 2022-03-17T20:15:09.1825211Z ---------------------------------------------------------------------- 2022-03-17T20:15:09.1846822Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:09.1847347Z 2022-03-17T20:15:09.1847991Z ---------------------------------------------------------------------- 2022-03-17T20:15:09.1848598Z Ran 1 test in 0.002s 2022-03-17T20:15:09.1848746Z 2022-03-17T20:15:09.1848855Z OK (skipped=1) 2022-03-17T20:15:09.1849007Z 2022-03-17T20:15:09.1849129Z Generating XML reports... 2022-03-17T20:15:09.1892792Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201509.xml 2022-03-17T20:15:10.3150913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:10.3166064Z 2022-03-17T20:15:10.3166290Z Running tests... 2022-03-17T20:15:10.3167214Z ---------------------------------------------------------------------- 2022-03-17T20:15:11.8454707Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:15:11.8874151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113353 2022-03-17T20:15:11.8998683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113354 2022-03-17T20:15:12.9542300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:12.9820961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:12.9822594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:12.9845743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:12.9852265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:13.0837440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:15.7087637Z ok (5.392s) 2022-03-17T20:15:15.7087996Z 2022-03-17T20:15:15.7088783Z ---------------------------------------------------------------------- 2022-03-17T20:15:15.7089619Z Ran 1 test in 5.392s 2022-03-17T20:15:15.7089768Z 2022-03-17T20:15:15.7089858Z OK 2022-03-17T20:15:15.7089996Z 2022-03-17T20:15:15.7090138Z Generating XML reports... 2022-03-17T20:15:15.7135023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201510.xml 2022-03-17T20:15:17.0158017Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:17.0173853Z 2022-03-17T20:15:17.0174171Z Running tests... 2022-03-17T20:15:17.0175046Z ---------------------------------------------------------------------- 2022-03-17T20:15:17.0196633Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:17.0197128Z 2022-03-17T20:15:17.0197774Z ---------------------------------------------------------------------- 2022-03-17T20:15:17.0198420Z Ran 1 test in 0.002s 2022-03-17T20:15:17.0198653Z 2022-03-17T20:15:17.0198746Z OK (skipped=1) 2022-03-17T20:15:17.0198911Z 2022-03-17T20:15:17.0199019Z Generating XML reports... 2022-03-17T20:15:17.0242085Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201517.xml 2022-03-17T20:15:18.1608277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:18.1623358Z 2022-03-17T20:15:18.1623838Z Running tests... 2022-03-17T20:15:18.1624329Z ---------------------------------------------------------------------- 2022-03-17T20:15:18.1645348Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:18.1645630Z 2022-03-17T20:15:18.1645917Z ---------------------------------------------------------------------- 2022-03-17T20:15:18.1646250Z Ran 1 test in 0.002s 2022-03-17T20:15:18.1646414Z 2022-03-17T20:15:18.1646522Z OK (skipped=1) 2022-03-17T20:15:18.1646659Z 2022-03-17T20:15:18.1646783Z Generating XML reports... 2022-03-17T20:15:18.1690631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201518.xml 2022-03-17T20:15:19.2755420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:19.2770834Z 2022-03-17T20:15:19.2771269Z Running tests... 2022-03-17T20:15:19.2771744Z ---------------------------------------------------------------------- 2022-03-17T20:15:19.2794452Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:19.2794734Z 2022-03-17T20:15:19.2795025Z ---------------------------------------------------------------------- 2022-03-17T20:15:19.2795338Z Ran 1 test in 0.002s 2022-03-17T20:15:19.2795518Z 2022-03-17T20:15:19.2795628Z OK (skipped=1) 2022-03-17T20:15:19.2795783Z 2022-03-17T20:15:19.2795893Z Generating XML reports... 2022-03-17T20:15:19.2839297Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201519.xml 2022-03-17T20:15:20.4353862Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:20.4369192Z 2022-03-17T20:15:20.4369686Z Running tests... 2022-03-17T20:15:20.4370300Z ---------------------------------------------------------------------- 2022-03-17T20:15:21.9921434Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:15:22.0345491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113581 2022-03-17T20:15:22.0470451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113582 2022-03-17T20:15:23.1102980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:23.1471982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:23.1472781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:23.1508193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:23.1515336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:23.2487350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:25.3555329Z ok (4.918s) 2022-03-17T20:15:25.3555579Z 2022-03-17T20:15:25.3556034Z ---------------------------------------------------------------------- 2022-03-17T20:15:25.3556371Z Ran 1 test in 4.919s 2022-03-17T20:15:25.3556519Z 2022-03-17T20:15:25.3556561Z OK 2022-03-17T20:15:25.3557004Z 2022-03-17T20:15:25.3557163Z Generating XML reports... 2022-03-17T20:15:25.3602967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201520.xml 2022-03-17T20:15:26.6606388Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:26.6622001Z 2022-03-17T20:15:26.6622347Z Running tests... 2022-03-17T20:15:26.6622773Z ---------------------------------------------------------------------- 2022-03-17T20:15:28.1936667Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:15:28.2361249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113707 2022-03-17T20:15:28.2485605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113708 2022-03-17T20:15:29.2950308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:29.2953239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:29.2954225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:29.3051861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:29.3058844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:29.3968404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:31.4578851Z ok (4.795s) 2022-03-17T20:15:31.4579055Z 2022-03-17T20:15:31.4579427Z ---------------------------------------------------------------------- 2022-03-17T20:15:31.4579772Z Ran 1 test in 4.796s 2022-03-17T20:15:31.4579940Z 2022-03-17T20:15:31.4580026Z OK 2022-03-17T20:15:31.4580141Z 2022-03-17T20:15:31.4580570Z Generating XML reports... 2022-03-17T20:15:31.4625332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201526.xml 2022-03-17T20:15:32.7609134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:32.7624661Z 2022-03-17T20:15:32.7625293Z Running tests... 2022-03-17T20:15:32.7625779Z ---------------------------------------------------------------------- 2022-03-17T20:15:34.3062370Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:15:34.3483373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113833 2022-03-17T20:15:34.3606045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113834 2022-03-17T20:15:35.4366807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:35.4488883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:35.4489679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:35.4570766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:35.4577472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:35.5504116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:37.6688473Z ok (4.906s) 2022-03-17T20:15:37.6688650Z 2022-03-17T20:15:37.6689119Z ---------------------------------------------------------------------- 2022-03-17T20:15:37.6689446Z Ran 1 test in 4.906s 2022-03-17T20:15:37.6689580Z 2022-03-17T20:15:37.6689653Z OK 2022-03-17T20:15:37.6689775Z 2022-03-17T20:15:37.6689909Z Generating XML reports... 2022-03-17T20:15:37.6736045Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201532.xml 2022-03-17T20:15:38.9676065Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:38.9691471Z 2022-03-17T20:15:38.9691625Z Running tests... 2022-03-17T20:15:38.9692091Z ---------------------------------------------------------------------- 2022-03-17T20:15:38.9711550Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:38.9711810Z 2022-03-17T20:15:38.9712108Z ---------------------------------------------------------------------- 2022-03-17T20:15:38.9712661Z Ran 1 test in 0.002s 2022-03-17T20:15:38.9712809Z 2022-03-17T20:15:38.9712915Z OK (skipped=1) 2022-03-17T20:15:38.9713071Z 2022-03-17T20:15:38.9713176Z Generating XML reports... 2022-03-17T20:15:38.9756461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201538.xml 2022-03-17T20:15:40.1013833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:40.1028688Z 2022-03-17T20:15:40.1028917Z Running tests... 2022-03-17T20:15:40.1029353Z ---------------------------------------------------------------------- 2022-03-17T20:15:40.1048407Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:40.1048663Z 2022-03-17T20:15:40.1049065Z ---------------------------------------------------------------------- 2022-03-17T20:15:40.1049353Z Ran 1 test in 0.002s 2022-03-17T20:15:40.1049498Z 2022-03-17T20:15:40.1049606Z OK (skipped=1) 2022-03-17T20:15:40.1049745Z 2022-03-17T20:15:40.1049877Z Generating XML reports... 2022-03-17T20:15:40.1092867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201540.xml 2022-03-17T20:15:41.2497132Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:41.2512816Z 2022-03-17T20:15:41.2513087Z Running tests... 2022-03-17T20:15:41.2513658Z ---------------------------------------------------------------------- 2022-03-17T20:15:42.7744765Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:15:42.8168629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114029 2022-03-17T20:15:42.8292412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114030 2022-03-17T20:15:43.8669548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:43.8980593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:43.8981646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:43.9075753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:43.9082225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:43.9995023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:46.6384012Z ok (5.387s) 2022-03-17T20:15:46.6384223Z 2022-03-17T20:15:46.6384722Z ---------------------------------------------------------------------- 2022-03-17T20:15:46.6385191Z Ran 1 test in 5.387s 2022-03-17T20:15:46.6385359Z 2022-03-17T20:15:46.6385515Z OK 2022-03-17T20:15:46.6385753Z 2022-03-17T20:15:46.6385961Z Generating XML reports... 2022-03-17T20:15:46.6430886Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201541.xml 2022-03-17T20:15:47.9356316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:47.9371222Z 2022-03-17T20:15:47.9371599Z Running tests... 2022-03-17T20:15:47.9372337Z ---------------------------------------------------------------------- 2022-03-17T20:15:49.4717950Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:15:49.5136385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114152 2022-03-17T20:15:49.5261894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114153 2022-03-17T20:15:50.5515038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:50.5637183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:50.5638464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:50.5718581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:50.5725106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:50.6651839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:53.3354588Z ok (5.398s) 2022-03-17T20:15:53.3354932Z 2022-03-17T20:15:53.3355718Z ---------------------------------------------------------------------- 2022-03-17T20:15:53.3356295Z Ran 1 test in 5.398s 2022-03-17T20:15:53.3356467Z 2022-03-17T20:15:53.3356558Z OK 2022-03-17T20:15:53.3356702Z 2022-03-17T20:15:53.3356835Z Generating XML reports... 2022-03-17T20:15:53.3399289Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201547.xml 2022-03-17T20:15:54.6283374Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:54.6298195Z 2022-03-17T20:15:54.6298505Z Running tests... 2022-03-17T20:15:54.6299456Z ---------------------------------------------------------------------- 2022-03-17T20:15:54.6318726Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:15:54.6318954Z 2022-03-17T20:15:54.6319277Z ---------------------------------------------------------------------- 2022-03-17T20:15:54.6319566Z Ran 1 test in 0.002s 2022-03-17T20:15:54.6319730Z 2022-03-17T20:15:54.6319851Z OK (skipped=1) 2022-03-17T20:15:54.6319985Z 2022-03-17T20:15:54.6320107Z Generating XML reports... 2022-03-17T20:15:54.6363717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201554.xml 2022-03-17T20:15:55.7757257Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:15:55.7772840Z 2022-03-17T20:15:55.7772951Z Running tests... 2022-03-17T20:15:55.7773409Z ---------------------------------------------------------------------- 2022-03-17T20:15:57.3286160Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:15:57.3710514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114310 2022-03-17T20:15:57.3834164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114311 2022-03-17T20:15:58.4718626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:15:58.4959122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:15:58.4960592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:58.5021957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:15:58.5028800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:15:58.5033257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:15:58.5973391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:15:58.5978918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:15:58.5980675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:15:58.6053655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:16:01.1923781Z ok (5.415s) 2022-03-17T20:16:01.1924134Z 2022-03-17T20:16:01.1924848Z ---------------------------------------------------------------------- 2022-03-17T20:16:01.1925303Z Ran 1 test in 5.415s 2022-03-17T20:16:01.1925469Z 2022-03-17T20:16:01.1925570Z OK 2022-03-17T20:16:01.1925713Z 2022-03-17T20:16:01.1925820Z Generating XML reports... 2022-03-17T20:16:01.1969215Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201555.xml 2022-03-17T20:16:02.4827797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:02.4842204Z 2022-03-17T20:16:02.4842676Z Running tests... 2022-03-17T20:16:02.4843612Z ---------------------------------------------------------------------- 2022-03-17T20:16:02.4862749Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:02.4863187Z 2022-03-17T20:16:02.4863757Z ---------------------------------------------------------------------- 2022-03-17T20:16:02.4864445Z Ran 1 test in 0.002s 2022-03-17T20:16:02.4864702Z 2022-03-17T20:16:02.4864850Z OK (skipped=1) 2022-03-17T20:16:02.4864989Z 2022-03-17T20:16:02.4865117Z Generating XML reports... 2022-03-17T20:16:02.4907716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201602.xml 2022-03-17T20:16:03.6387192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:03.6401460Z 2022-03-17T20:16:03.6401982Z Running tests... 2022-03-17T20:16:03.6402811Z ---------------------------------------------------------------------- 2022-03-17T20:16:05.1873677Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:16:05.2308621Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114472 2022-03-17T20:16:05.2435493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114473 2022-03-17T20:16:06.2829388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:06.2851731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:06.2852564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:06.2931356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:06.2938288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:06.3867528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:06.5480283Z skip (2.907s) 2022-03-17T20:16:06.5480527Z 2022-03-17T20:16:06.5480923Z ---------------------------------------------------------------------- 2022-03-17T20:16:06.5481240Z Ran 1 test in 2.908s 2022-03-17T20:16:06.5481410Z 2022-03-17T20:16:06.5481516Z OK (skipped=1) 2022-03-17T20:16:06.5481673Z 2022-03-17T20:16:06.5481796Z Generating XML reports... 2022-03-17T20:16:06.5526749Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201603.xml 2022-03-17T20:16:07.8355678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:07.8371026Z 2022-03-17T20:16:07.8371177Z Running tests... 2022-03-17T20:16:07.8371965Z ---------------------------------------------------------------------- 2022-03-17T20:16:07.8391582Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:07.8391880Z 2022-03-17T20:16:07.8392173Z ---------------------------------------------------------------------- 2022-03-17T20:16:07.8392503Z Ran 1 test in 0.002s 2022-03-17T20:16:07.8392653Z 2022-03-17T20:16:07.8392739Z OK (skipped=1) 2022-03-17T20:16:07.8392895Z 2022-03-17T20:16:07.8393005Z Generating XML reports... 2022-03-17T20:16:07.8436562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201607.xml 2022-03-17T20:16:08.9793447Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:08.9807812Z 2022-03-17T20:16:08.9808301Z Running tests... 2022-03-17T20:16:08.9808800Z ---------------------------------------------------------------------- 2022-03-17T20:16:08.9828500Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:08.9828789Z 2022-03-17T20:16:08.9829106Z ---------------------------------------------------------------------- 2022-03-17T20:16:08.9829405Z Ran 1 test in 0.002s 2022-03-17T20:16:08.9829567Z 2022-03-17T20:16:08.9829670Z OK (skipped=1) 2022-03-17T20:16:08.9829784Z 2022-03-17T20:16:08.9829911Z Generating XML reports... 2022-03-17T20:16:08.9873382Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201608.xml 2022-03-17T20:16:10.1270569Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:10.1285651Z 2022-03-17T20:16:10.1285990Z Running tests... 2022-03-17T20:16:10.1286430Z ---------------------------------------------------------------------- 2022-03-17T20:16:11.6813391Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:16:11.7236060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114655 2022-03-17T20:16:11.7362641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114656 2022-03-17T20:16:12.8246263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:12.8676099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:12.8676889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:12.8752037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:12.8758772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:12.9691314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:16.9479477Z ok (6.819s) 2022-03-17T20:16:16.9480249Z 2022-03-17T20:16:16.9480648Z ---------------------------------------------------------------------- 2022-03-17T20:16:16.9481327Z Ran 1 test in 6.819s 2022-03-17T20:16:16.9481473Z 2022-03-17T20:16:16.9481566Z OK 2022-03-17T20:16:16.9481682Z 2022-03-17T20:16:16.9481807Z Generating XML reports... 2022-03-17T20:16:16.9525726Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201610.xml 2022-03-17T20:16:18.2502270Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:18.2518771Z 2022-03-17T20:16:18.2518942Z Running tests... 2022-03-17T20:16:18.2519404Z ---------------------------------------------------------------------- 2022-03-17T20:16:19.8048196Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:16:19.8466611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114783 2022-03-17T20:16:19.8590405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114784 2022-03-17T20:16:20.9358074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:20.9492250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:20.9493081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:20.9561474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:20.9569486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:21.0508477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:24.9705087Z ok (6.718s) 2022-03-17T20:16:24.9705506Z 2022-03-17T20:16:24.9705910Z ---------------------------------------------------------------------- 2022-03-17T20:16:24.9706264Z Ran 1 test in 6.719s 2022-03-17T20:16:24.9706399Z 2022-03-17T20:16:24.9706495Z OK 2022-03-17T20:16:24.9706612Z 2022-03-17T20:16:24.9706747Z Generating XML reports... 2022-03-17T20:16:24.9752668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201618.xml 2022-03-17T20:16:26.2678814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:26.2693693Z 2022-03-17T20:16:26.2694119Z Running tests... 2022-03-17T20:16:26.2694612Z ---------------------------------------------------------------------- 2022-03-17T20:16:26.2714218Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:26.2714554Z 2022-03-17T20:16:26.2714828Z ---------------------------------------------------------------------- 2022-03-17T20:16:26.2715156Z Ran 1 test in 0.002s 2022-03-17T20:16:26.2715316Z 2022-03-17T20:16:26.2715402Z OK (skipped=1) 2022-03-17T20:16:26.2715550Z 2022-03-17T20:16:26.2715674Z Generating XML reports... 2022-03-17T20:16:26.2759170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201626.xml 2022-03-17T20:16:27.4143680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:27.4158652Z 2022-03-17T20:16:27.4158909Z Running tests... 2022-03-17T20:16:27.4159856Z ---------------------------------------------------------------------- 2022-03-17T20:16:28.9601514Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:16:29.0025483Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114946 2022-03-17T20:16:29.0151347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114947 2022-03-17T20:16:30.0741621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:30.0824913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:30.0825711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:30.0842679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:30.0850049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:30.0853484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:16:30.1839567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:30.1844194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:16:30.1844950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:16:30.1872879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:16:34.2268649Z ok (6.811s) 2022-03-17T20:16:34.2268898Z 2022-03-17T20:16:34.2269301Z ---------------------------------------------------------------------- 2022-03-17T20:16:34.2269647Z Ran 1 test in 6.811s 2022-03-17T20:16:34.2269794Z 2022-03-17T20:16:34.2269887Z OK 2022-03-17T20:16:34.2270024Z 2022-03-17T20:16:34.2270158Z Generating XML reports... 2022-03-17T20:16:34.2315439Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201627.xml 2022-03-17T20:16:35.5241245Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:35.5255924Z 2022-03-17T20:16:35.5256200Z Running tests... 2022-03-17T20:16:35.5256632Z ---------------------------------------------------------------------- 2022-03-17T20:16:35.5276075Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:35.5276368Z 2022-03-17T20:16:35.5276654Z ---------------------------------------------------------------------- 2022-03-17T20:16:35.5276960Z Ran 1 test in 0.002s 2022-03-17T20:16:35.5277122Z 2022-03-17T20:16:35.5277231Z OK (skipped=1) 2022-03-17T20:16:35.5277386Z 2022-03-17T20:16:35.5277493Z Generating XML reports... 2022-03-17T20:16:35.5321214Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201635.xml 2022-03-17T20:16:36.6712533Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:36.6727262Z 2022-03-17T20:16:36.6727475Z Running tests... 2022-03-17T20:16:36.6727896Z ---------------------------------------------------------------------- 2022-03-17T20:16:38.2187293Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:16:38.2614186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115113 2022-03-17T20:16:38.2741695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115114 2022-03-17T20:16:39.3243433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:39.3435204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:39.3435978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:39.3446473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:39.3452929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:39.4449663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:39.5787227Z skip (2.906s) 2022-03-17T20:16:39.5787451Z 2022-03-17T20:16:39.5787839Z ---------------------------------------------------------------------- 2022-03-17T20:16:39.5788154Z Ran 1 test in 2.906s 2022-03-17T20:16:39.5788319Z 2022-03-17T20:16:39.5788426Z OK (skipped=1) 2022-03-17T20:16:39.5788580Z 2022-03-17T20:16:39.5788686Z Generating XML reports... 2022-03-17T20:16:39.5833734Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201636.xml 2022-03-17T20:16:40.8657230Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:40.8673265Z 2022-03-17T20:16:40.8673813Z Running tests... 2022-03-17T20:16:40.8674233Z ---------------------------------------------------------------------- 2022-03-17T20:16:40.8693658Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:40.8693960Z 2022-03-17T20:16:40.8694255Z ---------------------------------------------------------------------- 2022-03-17T20:16:40.8694585Z Ran 1 test in 0.002s 2022-03-17T20:16:40.8694732Z 2022-03-17T20:16:40.8694851Z OK (skipped=1) 2022-03-17T20:16:40.8695006Z 2022-03-17T20:16:40.8695112Z Generating XML reports... 2022-03-17T20:16:40.8739588Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201640.xml 2022-03-17T20:16:42.0243006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:42.0258270Z 2022-03-17T20:16:42.0258712Z Running tests... 2022-03-17T20:16:42.0259246Z ---------------------------------------------------------------------- 2022-03-17T20:16:42.0279773Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:42.0280084Z 2022-03-17T20:16:42.0280416Z ---------------------------------------------------------------------- 2022-03-17T20:16:42.0280769Z Ran 1 test in 0.002s 2022-03-17T20:16:42.0280953Z 2022-03-17T20:16:42.0281022Z OK (skipped=1) 2022-03-17T20:16:42.0281120Z 2022-03-17T20:16:42.0281248Z Generating XML reports... 2022-03-17T20:16:42.0325045Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201642.xml 2022-03-17T20:16:43.1773611Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:43.1789145Z 2022-03-17T20:16:43.1789376Z Running tests... 2022-03-17T20:16:43.1789844Z ---------------------------------------------------------------------- 2022-03-17T20:16:44.7350242Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:16:44.7790797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115296 2022-03-17T20:16:44.7920789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115297 2022-03-17T20:16:45.8588998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:45.8929377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:45.8930194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:45.8994168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:45.9000687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:45.9944940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:48.7012759Z ok (5.522s) 2022-03-17T20:16:48.7012984Z 2022-03-17T20:16:48.7013373Z ---------------------------------------------------------------------- 2022-03-17T20:16:48.7013691Z Ran 1 test in 5.522s 2022-03-17T20:16:48.7014204Z 2022-03-17T20:16:48.7014298Z OK 2022-03-17T20:16:48.7014373Z 2022-03-17T20:16:48.7014592Z Generating XML reports... 2022-03-17T20:16:48.7058013Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201643.xml 2022-03-17T20:16:50.0056011Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:50.0072495Z 2022-03-17T20:16:50.0072741Z Running tests... 2022-03-17T20:16:50.0073244Z ---------------------------------------------------------------------- 2022-03-17T20:16:51.5986727Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:16:51.6418913Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115419 2022-03-17T20:16:51.6544086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115420 2022-03-17T20:16:52.6957762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:16:52.7286873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:16:52.7287694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:52.7362518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:16:52.7369049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:16:52.8301730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:16:55.4650642Z ok (5.457s) 2022-03-17T20:16:55.4650939Z 2022-03-17T20:16:55.4651261Z ---------------------------------------------------------------------- 2022-03-17T20:16:55.4651586Z Ran 1 test in 5.458s 2022-03-17T20:16:55.4651778Z 2022-03-17T20:16:55.4651870Z OK 2022-03-17T20:16:55.4652003Z 2022-03-17T20:16:55.4652169Z Generating XML reports... 2022-03-17T20:16:55.4696702Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201649.xml 2022-03-17T20:16:56.7280394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:56.7295757Z 2022-03-17T20:16:56.7295906Z Running tests... 2022-03-17T20:16:56.7296339Z ---------------------------------------------------------------------- 2022-03-17T20:16:56.7316434Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:16:56.7316979Z 2022-03-17T20:16:56.7317430Z ---------------------------------------------------------------------- 2022-03-17T20:16:56.7317830Z Ran 1 test in 0.002s 2022-03-17T20:16:56.7318004Z 2022-03-17T20:16:56.7318107Z OK (skipped=1) 2022-03-17T20:16:56.7318266Z 2022-03-17T20:16:56.7318445Z Generating XML reports... 2022-03-17T20:16:56.7362312Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201656.xml 2022-03-17T20:16:57.8763586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:16:57.8778630Z 2022-03-17T20:16:57.8778908Z Running tests... 2022-03-17T20:16:57.8779405Z ---------------------------------------------------------------------- 2022-03-17T20:16:59.4027101Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:16:59.4458456Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115577 2022-03-17T20:16:59.4588583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115578 2022-03-17T20:17:00.5387291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:00.5690294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:00.5691349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:00.5691947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:00.5699146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:00.5699841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:00.5704735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:17:00.5705272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:17:00.5705930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:17:00.5706599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:17:03.2680543Z ok (5.390s) 2022-03-17T20:17:03.2680765Z 2022-03-17T20:17:03.2681159Z ---------------------------------------------------------------------- 2022-03-17T20:17:03.2681493Z Ran 1 test in 5.390s 2022-03-17T20:17:03.2681664Z 2022-03-17T20:17:03.2681745Z OK 2022-03-17T20:17:03.2681869Z 2022-03-17T20:17:03.2681992Z Generating XML reports... 2022-03-17T20:17:03.2727299Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201657.xml 2022-03-17T20:17:04.5642377Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:04.5657624Z 2022-03-17T20:17:04.5657936Z Running tests... 2022-03-17T20:17:04.5658461Z ---------------------------------------------------------------------- 2022-03-17T20:17:04.5677876Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:04.5678354Z 2022-03-17T20:17:04.5678794Z ---------------------------------------------------------------------- 2022-03-17T20:17:04.5679106Z Ran 1 test in 0.002s 2022-03-17T20:17:04.5679232Z 2022-03-17T20:17:04.5679341Z OK (skipped=1) 2022-03-17T20:17:04.5679494Z 2022-03-17T20:17:04.5679639Z Generating XML reports... 2022-03-17T20:17:04.5722910Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201704.xml 2022-03-17T20:17:05.7191016Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:05.7206549Z 2022-03-17T20:17:05.7206710Z Running tests... 2022-03-17T20:17:05.7207596Z ---------------------------------------------------------------------- 2022-03-17T20:17:07.2680648Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:07.3103516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115739 2022-03-17T20:17:07.3230005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115740 2022-03-17T20:17:08.3627861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:08.3970811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:08.3971694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:08.4033071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:08.4039691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:08.4986137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:08.6278103Z skip (2.907s) 2022-03-17T20:17:08.6278357Z 2022-03-17T20:17:08.6278766Z ---------------------------------------------------------------------- 2022-03-17T20:17:08.6279107Z Ran 1 test in 2.907s 2022-03-17T20:17:08.6279184Z 2022-03-17T20:17:08.6279297Z OK (skipped=1) 2022-03-17T20:17:08.6279435Z 2022-03-17T20:17:08.6279565Z Generating XML reports... 2022-03-17T20:17:08.6324411Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201705.xml 2022-03-17T20:17:09.9053718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:09.9069516Z 2022-03-17T20:17:09.9070058Z Running tests... 2022-03-17T20:17:09.9070486Z ---------------------------------------------------------------------- 2022-03-17T20:17:11.4242258Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:11.4659877Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115852 2022-03-17T20:17:11.4784546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115853 2022-03-17T20:17:12.5468798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:12.5551445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:12.5552229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:12.5570161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:12.5576857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:12.6566173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:14.9094751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:17:14.9095341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:17:14.9096106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:17:14.9096808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:17:15.2876708Z ok (5.380s) 2022-03-17T20:17:15.2876892Z 2022-03-17T20:17:15.2877513Z ---------------------------------------------------------------------- 2022-03-17T20:17:15.2877886Z Ran 1 test in 5.381s 2022-03-17T20:17:15.2878062Z 2022-03-17T20:17:15.2878154Z OK 2022-03-17T20:17:15.2878271Z 2022-03-17T20:17:15.2878400Z Generating XML reports... 2022-03-17T20:17:15.2922727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201709.xml 2022-03-17T20:17:16.5838062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:16.5852777Z 2022-03-17T20:17:16.5853029Z Running tests... 2022-03-17T20:17:16.5853472Z ---------------------------------------------------------------------- 2022-03-17T20:17:18.1182182Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:18.1603768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115986 2022-03-17T20:17:18.1727295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115987 2022-03-17T20:17:19.2093863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:19.2147227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:19.2148000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:19.2197621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:19.2204442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:19.3161171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:19.4775484Z skip (2.892s) 2022-03-17T20:17:19.4776105Z 2022-03-17T20:17:19.4776644Z ---------------------------------------------------------------------- 2022-03-17T20:17:19.4776992Z Ran 1 test in 2.892s 2022-03-17T20:17:19.4777137Z 2022-03-17T20:17:19.4777515Z OK (skipped=1) 2022-03-17T20:17:19.4777613Z 2022-03-17T20:17:19.4777748Z Generating XML reports... 2022-03-17T20:17:19.4821659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201716.xml 2022-03-17T20:17:20.7649464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:20.7664750Z 2022-03-17T20:17:20.7665198Z Running tests... 2022-03-17T20:17:20.7665653Z ---------------------------------------------------------------------- 2022-03-17T20:17:20.7685689Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:20.7686001Z 2022-03-17T20:17:20.7686335Z ---------------------------------------------------------------------- 2022-03-17T20:17:20.7686669Z Ran 1 test in 0.002s 2022-03-17T20:17:20.7686833Z 2022-03-17T20:17:20.7686927Z OK (skipped=1) 2022-03-17T20:17:20.7687016Z 2022-03-17T20:17:20.7687141Z Generating XML reports... 2022-03-17T20:17:20.7730482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201720.xml 2022-03-17T20:17:21.9213109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:21.9228535Z 2022-03-17T20:17:21.9228721Z Running tests... 2022-03-17T20:17:21.9229206Z ---------------------------------------------------------------------- 2022-03-17T20:17:21.9248533Z test_barrier (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:21.9248796Z 2022-03-17T20:17:21.9249094Z ---------------------------------------------------------------------- 2022-03-17T20:17:21.9249434Z Ran 1 test in 0.002s 2022-03-17T20:17:21.9249570Z 2022-03-17T20:17:21.9249693Z OK (skipped=1) 2022-03-17T20:17:21.9249847Z 2022-03-17T20:17:21.9249964Z Generating XML reports... 2022-03-17T20:17:21.9293387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201721.xml 2022-03-17T20:17:23.0148814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:23.0163783Z 2022-03-17T20:17:23.0164069Z Running tests... 2022-03-17T20:17:23.0164514Z ---------------------------------------------------------------------- 2022-03-17T20:17:24.5372463Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:24.5792182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116169 2022-03-17T20:17:24.5917728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116170 2022-03-17T20:17:25.6863633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:25.6955482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:25.6956288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:25.6965567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:25.6972138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:25.7971282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:29.2021625Z ok (6.185s) 2022-03-17T20:17:29.2021806Z 2022-03-17T20:17:29.2022244Z ---------------------------------------------------------------------- 2022-03-17T20:17:29.2022567Z Ran 1 test in 6.186s 2022-03-17T20:17:29.2022740Z 2022-03-17T20:17:29.2022806Z OK 2022-03-17T20:17:29.2022949Z 2022-03-17T20:17:29.2023442Z Generating XML reports... 2022-03-17T20:17:29.2070314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201723.xml 2022-03-17T20:17:30.4895136Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:30.4910281Z 2022-03-17T20:17:30.4910723Z Running tests... 2022-03-17T20:17:30.4911233Z ---------------------------------------------------------------------- 2022-03-17T20:17:30.4929560Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:30.4929907Z 2022-03-17T20:17:30.4930259Z ---------------------------------------------------------------------- 2022-03-17T20:17:30.4930590Z Ran 1 test in 0.002s 2022-03-17T20:17:30.4930755Z 2022-03-17T20:17:30.4930877Z OK (skipped=1) 2022-03-17T20:17:30.4931014Z 2022-03-17T20:17:30.4931137Z Generating XML reports... 2022-03-17T20:17:30.4974036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201730.xml 2022-03-17T20:17:31.6366884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:31.6382397Z 2022-03-17T20:17:31.6382731Z Running tests... 2022-03-17T20:17:31.6383197Z ---------------------------------------------------------------------- 2022-03-17T20:17:33.1684760Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:33.2108060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116327 2022-03-17T20:17:33.2233200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116328 2022-03-17T20:17:34.3212018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:34.3264882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:34.3265689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:34.3313581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:34.3320093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:34.4278687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:34.5278064Z skip (2.889s) 2022-03-17T20:17:34.5278305Z 2022-03-17T20:17:34.5278694Z ---------------------------------------------------------------------- 2022-03-17T20:17:34.5279012Z Ran 1 test in 2.890s 2022-03-17T20:17:34.5279179Z 2022-03-17T20:17:34.5279293Z OK (skipped=1) 2022-03-17T20:17:34.5279449Z 2022-03-17T20:17:34.5279557Z Generating XML reports... 2022-03-17T20:17:34.5324181Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201731.xml 2022-03-17T20:17:35.7673185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:35.7690656Z 2022-03-17T20:17:35.7691110Z Running tests... 2022-03-17T20:17:35.7691640Z ---------------------------------------------------------------------- 2022-03-17T20:17:35.7711922Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:35.7712175Z 2022-03-17T20:17:35.7712795Z ---------------------------------------------------------------------- 2022-03-17T20:17:35.7713430Z Ran 1 test in 0.002s 2022-03-17T20:17:35.7713596Z 2022-03-17T20:17:35.7713722Z OK (skipped=1) 2022-03-17T20:17:35.7713860Z 2022-03-17T20:17:35.7713990Z Generating XML reports... 2022-03-17T20:17:35.7758335Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201735.xml 2022-03-17T20:17:36.9168123Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:36.9184882Z 2022-03-17T20:17:36.9185325Z Running tests... 2022-03-17T20:17:36.9185801Z ---------------------------------------------------------------------- 2022-03-17T20:17:38.4839733Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:38.5258711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116475 2022-03-17T20:17:38.5384328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116476 2022-03-17T20:17:39.5735088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:39.5769203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:39.5770234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:39.5836781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:39.5843237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:39.6784412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:39.8433372Z skip (2.925s) 2022-03-17T20:17:39.8433652Z 2022-03-17T20:17:39.8434023Z ---------------------------------------------------------------------- 2022-03-17T20:17:39.8434337Z Ran 1 test in 2.925s 2022-03-17T20:17:39.8434493Z 2022-03-17T20:17:39.8434618Z OK (skipped=1) 2022-03-17T20:17:39.8434789Z 2022-03-17T20:17:39.8434895Z Generating XML reports... 2022-03-17T20:17:39.8479747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201736.xml 2022-03-17T20:17:41.1217179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:41.1232576Z 2022-03-17T20:17:41.1233019Z Running tests... 2022-03-17T20:17:41.1233965Z ---------------------------------------------------------------------- 2022-03-17T20:17:41.1253967Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:41.1254467Z 2022-03-17T20:17:41.1255036Z ---------------------------------------------------------------------- 2022-03-17T20:17:41.1255439Z Ran 1 test in 0.002s 2022-03-17T20:17:41.1255739Z 2022-03-17T20:17:41.1255903Z OK (skipped=1) 2022-03-17T20:17:41.1256225Z 2022-03-17T20:17:41.1256502Z Generating XML reports... 2022-03-17T20:17:41.1298636Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201741.xml 2022-03-17T20:17:42.2768529Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:42.2784027Z 2022-03-17T20:17:42.2784364Z Running tests... 2022-03-17T20:17:42.2784810Z ---------------------------------------------------------------------- 2022-03-17T20:17:42.2809122Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:42.2809412Z 2022-03-17T20:17:42.2809739Z ---------------------------------------------------------------------- 2022-03-17T20:17:42.2810079Z Ran 1 test in 0.003s 2022-03-17T20:17:42.2810223Z 2022-03-17T20:17:42.2810310Z OK (skipped=1) 2022-03-17T20:17:42.2810445Z 2022-03-17T20:17:42.2810568Z Generating XML reports... 2022-03-17T20:17:42.2854962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201742.xml 2022-03-17T20:17:43.4144456Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:43.4160271Z 2022-03-17T20:17:43.4160377Z Running tests... 2022-03-17T20:17:43.4161405Z ---------------------------------------------------------------------- 2022-03-17T20:17:43.4182276Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:17:43.4182683Z 2022-03-17T20:17:43.4182924Z ---------------------------------------------------------------------- 2022-03-17T20:17:43.4183236Z Ran 1 test in 0.002s 2022-03-17T20:17:43.4183451Z 2022-03-17T20:17:43.4183663Z OK (skipped=1) 2022-03-17T20:17:43.4183952Z 2022-03-17T20:17:43.4184362Z Generating XML reports... 2022-03-17T20:17:43.4226890Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201743.xml 2022-03-17T20:17:44.5621041Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:44.5636185Z 2022-03-17T20:17:44.5636726Z Running tests... 2022-03-17T20:17:44.5637325Z ---------------------------------------------------------------------- 2022-03-17T20:17:44.5663450Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:17:44.5663956Z 2022-03-17T20:17:44.5664556Z ---------------------------------------------------------------------- 2022-03-17T20:17:44.5665161Z Ran 1 test in 0.003s 2022-03-17T20:17:44.5665307Z 2022-03-17T20:17:44.5665421Z OK (skipped=1) 2022-03-17T20:17:44.5665576Z 2022-03-17T20:17:44.5665700Z Generating XML reports... 2022-03-17T20:17:44.5709375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201744.xml 2022-03-17T20:17:45.7047506Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:45.7062845Z 2022-03-17T20:17:45.7063046Z Running tests... 2022-03-17T20:17:45.7064241Z ---------------------------------------------------------------------- 2022-03-17T20:17:45.7090234Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:17:45.7090751Z 2022-03-17T20:17:45.7091264Z ---------------------------------------------------------------------- 2022-03-17T20:17:45.7092007Z Ran 1 test in 0.003s 2022-03-17T20:17:45.7092321Z 2022-03-17T20:17:45.7092456Z OK (skipped=1) 2022-03-17T20:17:45.7092616Z 2022-03-17T20:17:45.7092743Z Generating XML reports... 2022-03-17T20:17:45.7135942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201745.xml 2022-03-17T20:17:46.8046185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:46.8061234Z 2022-03-17T20:17:46.8061484Z Running tests... 2022-03-17T20:17:46.8061929Z ---------------------------------------------------------------------- 2022-03-17T20:17:48.3689821Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:48.4108311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116763 2022-03-17T20:17:48.4237808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116764 2022-03-17T20:17:49.4582342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:49.4648056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:49.4648887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:49.4685140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:49.4691748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:49.5659640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:49.5888642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:17:49.5889158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:17:49.5889836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:17:49.5890530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:17:49.5893288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:17:49.5996363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:17:49.5997049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:17:49.6098400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:17:49.8284926Z ok (3.022s) 2022-03-17T20:17:49.8285144Z 2022-03-17T20:17:49.8285551Z ---------------------------------------------------------------------- 2022-03-17T20:17:49.8285871Z Ran 1 test in 3.022s 2022-03-17T20:17:49.8286034Z 2022-03-17T20:17:49.8286134Z OK 2022-03-17T20:17:49.8286272Z 2022-03-17T20:17:49.8286384Z Generating XML reports... 2022-03-17T20:17:49.8331164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201746.xml 2022-03-17T20:17:51.1030160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:51.1045328Z 2022-03-17T20:17:51.1045772Z Running tests... 2022-03-17T20:17:51.1046300Z ---------------------------------------------------------------------- 2022-03-17T20:17:52.6476033Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:52.6897555Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116886 2022-03-17T20:17:52.7022664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116887 2022-03-17T20:17:53.7404143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:53.7870520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:53.7871327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:53.7911382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:53.7918083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:53.8881938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:55.6101739Z ok (4.505s) 2022-03-17T20:17:55.6101942Z 2022-03-17T20:17:55.6102342Z ---------------------------------------------------------------------- 2022-03-17T20:17:55.6102684Z Ran 1 test in 4.506s 2022-03-17T20:17:55.6102846Z 2022-03-17T20:17:55.6102887Z OK 2022-03-17T20:17:55.6103021Z 2022-03-17T20:17:55.6103154Z Generating XML reports... 2022-03-17T20:17:55.6147449Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201751.xml 2022-03-17T20:17:56.9114777Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:17:56.9129918Z 2022-03-17T20:17:56.9130336Z Running tests... 2022-03-17T20:17:56.9130826Z ---------------------------------------------------------------------- 2022-03-17T20:17:58.4573070Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:17:58.4984854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117016 2022-03-17T20:17:58.5110148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117017 2022-03-17T20:17:59.5443224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:17:59.5564389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:17:59.5565151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:59.5646692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:17:59.5653361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:17:59.6579118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:17:59.8158127Z skip (2.902s) 2022-03-17T20:17:59.8158397Z 2022-03-17T20:17:59.8158780Z ---------------------------------------------------------------------- 2022-03-17T20:17:59.8159111Z Ran 1 test in 2.903s 2022-03-17T20:17:59.8159261Z 2022-03-17T20:17:59.8159374Z OK (skipped=1) 2022-03-17T20:17:59.8159526Z 2022-03-17T20:17:59.8159636Z Generating XML reports... 2022-03-17T20:17:59.8205064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201756.xml 2022-03-17T20:18:01.0967126Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:01.0982870Z 2022-03-17T20:18:01.0983133Z Running tests... 2022-03-17T20:18:01.0983589Z ---------------------------------------------------------------------- 2022-03-17T20:18:02.6437420Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:02.6863310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117129 2022-03-17T20:18:02.6988448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117130 2022-03-17T20:18:03.7738078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:03.7978397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:03.7979204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:03.8042378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:03.8048610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:03.8993239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:05.4060393Z ok (4.307s) 2022-03-17T20:18:05.4060846Z 2022-03-17T20:18:05.4061260Z ---------------------------------------------------------------------- 2022-03-17T20:18:05.4061597Z Ran 1 test in 4.308s 2022-03-17T20:18:05.4061764Z 2022-03-17T20:18:05.4061842Z OK 2022-03-17T20:18:05.4061979Z 2022-03-17T20:18:05.4062119Z Generating XML reports... 2022-03-17T20:18:05.4107640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201801.xml 2022-03-17T20:18:06.7160837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:06.7176468Z 2022-03-17T20:18:06.7176959Z Running tests... 2022-03-17T20:18:06.7177385Z ---------------------------------------------------------------------- 2022-03-17T20:18:08.2666790Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:08.3090723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117243 2022-03-17T20:18:08.3213853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117244 2022-03-17T20:18:09.3565227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:09.3764165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:09.3764953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:09.3768379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:09.3775087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:09.4776543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:09.6259858Z ok (2.908s) 2022-03-17T20:18:09.6260086Z 2022-03-17T20:18:09.6260701Z ---------------------------------------------------------------------- 2022-03-17T20:18:09.6261049Z Ran 1 test in 2.908s 2022-03-17T20:18:09.6261220Z 2022-03-17T20:18:09.6261295Z OK 2022-03-17T20:18:09.6261435Z 2022-03-17T20:18:09.6261570Z Generating XML reports... 2022-03-17T20:18:09.6306824Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201806.xml 2022-03-17T20:18:10.9097321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:10.9113991Z 2022-03-17T20:18:10.9114296Z Running tests... 2022-03-17T20:18:10.9114764Z ---------------------------------------------------------------------- 2022-03-17T20:18:12.4595763Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:12.5018360Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117356 2022-03-17T20:18:12.5141933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117357 2022-03-17T20:18:13.5540946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:13.5548293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:13.5549119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:13.5644512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:13.5650959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:13.6560577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:15.3215872Z ok (4.410s) 2022-03-17T20:18:15.3216087Z 2022-03-17T20:18:15.3216468Z ---------------------------------------------------------------------- 2022-03-17T20:18:15.3216800Z Ran 1 test in 4.410s 2022-03-17T20:18:15.3216952Z 2022-03-17T20:18:15.3217030Z OK 2022-03-17T20:18:15.3217065Z 2022-03-17T20:18:15.3217197Z Generating XML reports... 2022-03-17T20:18:15.3261012Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201810.xml 2022-03-17T20:18:16.6177363Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:16.6192727Z 2022-03-17T20:18:16.6192876Z Running tests... 2022-03-17T20:18:16.6193574Z ---------------------------------------------------------------------- 2022-03-17T20:18:18.1639612Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:18.2064316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117474 2022-03-17T20:18:18.2189469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117475 2022-03-17T20:18:19.2604431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:19.2604961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:19.2605746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:19.2606420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:19.2613434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:19.2613938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:19.5235329Z ok (2.904s) 2022-03-17T20:18:19.5235536Z 2022-03-17T20:18:19.5235945Z ---------------------------------------------------------------------- 2022-03-17T20:18:19.5236309Z Ran 1 test in 2.904s 2022-03-17T20:18:19.5236456Z 2022-03-17T20:18:19.5236551Z OK 2022-03-17T20:18:19.5236689Z 2022-03-17T20:18:19.5236827Z Generating XML reports... 2022-03-17T20:18:19.5281293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201816.xml 2022-03-17T20:18:20.8084523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:20.8099665Z 2022-03-17T20:18:20.8099958Z Running tests... 2022-03-17T20:18:20.8100401Z ---------------------------------------------------------------------- 2022-03-17T20:18:20.8120370Z test_broadcast (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:18:20.8120622Z 2022-03-17T20:18:20.8120910Z ---------------------------------------------------------------------- 2022-03-17T20:18:20.8121217Z Ran 1 test in 0.002s 2022-03-17T20:18:20.8121382Z 2022-03-17T20:18:20.8121510Z OK (skipped=1) 2022-03-17T20:18:20.8121670Z 2022-03-17T20:18:20.8121799Z Generating XML reports... 2022-03-17T20:18:20.8165918Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201820.xml 2022-03-17T20:18:21.9603674Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:21.9618522Z 2022-03-17T20:18:21.9618956Z Running tests... 2022-03-17T20:18:21.9619472Z ---------------------------------------------------------------------- 2022-03-17T20:18:23.4987857Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:23.5410185Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117622 2022-03-17T20:18:23.5535576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117623 2022-03-17T20:18:24.6248801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:24.6519354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:24.6520173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:24.6552429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:24.6559143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:24.7535323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:26.8619453Z ok (4.900s) 2022-03-17T20:18:26.8619719Z 2022-03-17T20:18:26.8620142Z ---------------------------------------------------------------------- 2022-03-17T20:18:26.8620755Z Ran 1 test in 4.900s 2022-03-17T20:18:26.8620888Z 2022-03-17T20:18:26.8620963Z OK 2022-03-17T20:18:26.8621101Z 2022-03-17T20:18:26.8621241Z Generating XML reports... 2022-03-17T20:18:26.8665228Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201821.xml 2022-03-17T20:18:28.1689832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:28.1705646Z 2022-03-17T20:18:28.1705783Z Running tests... 2022-03-17T20:18:28.1706648Z ---------------------------------------------------------------------- 2022-03-17T20:18:28.1726416Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:18:28.1726676Z 2022-03-17T20:18:28.1727081Z ---------------------------------------------------------------------- 2022-03-17T20:18:28.1727606Z Ran 1 test in 0.002s 2022-03-17T20:18:28.1727755Z 2022-03-17T20:18:28.1727857Z OK (skipped=1) 2022-03-17T20:18:28.1728015Z 2022-03-17T20:18:28.1728141Z Generating XML reports... 2022-03-17T20:18:28.1771769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201828.xml 2022-03-17T20:18:29.3135599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:29.3150823Z 2022-03-17T20:18:29.3151260Z Running tests... 2022-03-17T20:18:29.3151655Z ---------------------------------------------------------------------- 2022-03-17T20:18:29.3171256Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:18:29.3171550Z 2022-03-17T20:18:29.3171867Z ---------------------------------------------------------------------- 2022-03-17T20:18:29.3172207Z Ran 1 test in 0.002s 2022-03-17T20:18:29.3172378Z 2022-03-17T20:18:29.3172497Z OK (skipped=1) 2022-03-17T20:18:29.3172647Z 2022-03-17T20:18:29.3172775Z Generating XML reports... 2022-03-17T20:18:29.3216328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201829.xml 2022-03-17T20:18:30.4570809Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:30.4586053Z 2022-03-17T20:18:30.4586183Z Running tests... 2022-03-17T20:18:30.4586955Z ---------------------------------------------------------------------- 2022-03-17T20:18:30.4607276Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:18:30.4608075Z 2022-03-17T20:18:30.4608357Z ---------------------------------------------------------------------- 2022-03-17T20:18:30.4608751Z Ran 1 test in 0.002s 2022-03-17T20:18:30.4608909Z 2022-03-17T20:18:30.4609024Z OK (skipped=1) 2022-03-17T20:18:30.4609158Z 2022-03-17T20:18:30.4609285Z Generating XML reports... 2022-03-17T20:18:30.4653058Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201830.xml 2022-03-17T20:18:31.6083459Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:31.6099257Z 2022-03-17T20:18:31.6099372Z Running tests... 2022-03-17T20:18:31.6101050Z ---------------------------------------------------------------------- 2022-03-17T20:18:33.1546377Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:33.1970881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117853 2022-03-17T20:18:33.2095673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117854 2022-03-17T20:18:34.2418663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:34.2623549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:34.2624352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:34.2723168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:34.2731052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:34.3639222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:37.0188626Z ok (5.409s) 2022-03-17T20:18:37.0188844Z 2022-03-17T20:18:37.0189262Z ---------------------------------------------------------------------- 2022-03-17T20:18:37.0189526Z Ran 1 test in 5.409s 2022-03-17T20:18:37.0189695Z 2022-03-17T20:18:37.0189794Z OK 2022-03-17T20:18:37.0189934Z 2022-03-17T20:18:37.0190067Z Generating XML reports... 2022-03-17T20:18:37.0235223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201831.xml 2022-03-17T20:18:38.3231239Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:38.3246357Z 2022-03-17T20:18:38.3246780Z Running tests... 2022-03-17T20:18:38.3247273Z ---------------------------------------------------------------------- 2022-03-17T20:18:39.8606385Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:39.9032333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117976 2022-03-17T20:18:39.9156199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117977 2022-03-17T20:18:40.9863187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:41.0164983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:41.0165787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:41.0166470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:41.0174599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:41.0175396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:41.0386950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:18:41.0387467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:18:41.0388164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:18:41.0388941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:18:41.0390523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:18:41.0391029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:18:41.0391687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:18:41.0392373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:18:42.7231483Z ok (4.398s) 2022-03-17T20:18:42.7231863Z 2022-03-17T20:18:42.7232468Z ---------------------------------------------------------------------- 2022-03-17T20:18:42.7233111Z Ran 1 test in 4.398s 2022-03-17T20:18:42.7233409Z 2022-03-17T20:18:42.7233558Z OK 2022-03-17T20:18:42.7233786Z 2022-03-17T20:18:42.7234325Z Generating XML reports... 2022-03-17T20:18:42.7278686Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201838.xml 2022-03-17T20:18:44.0226807Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:44.0242019Z 2022-03-17T20:18:44.0242231Z Running tests... 2022-03-17T20:18:44.0243048Z ---------------------------------------------------------------------- 2022-03-17T20:18:45.5506119Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:45.5929442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118106 2022-03-17T20:18:45.6053040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118107 2022-03-17T20:18:46.6932148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:46.7044941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:46.7045737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:46.7136860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:46.7143567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:46.8057079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:46.8167178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:18:46.8167705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:18:46.8168367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:18:46.8169050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:18:46.8170508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:18:46.8171457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:18:46.8172138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:18:46.8173165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:18:48.5128507Z ok (4.488s) 2022-03-17T20:18:48.5128741Z 2022-03-17T20:18:48.5129140Z ---------------------------------------------------------------------- 2022-03-17T20:18:48.5129462Z Ran 1 test in 4.489s 2022-03-17T20:18:48.5129630Z 2022-03-17T20:18:48.5129723Z OK 2022-03-17T20:18:48.5129859Z 2022-03-17T20:18:48.5130015Z Generating XML reports... 2022-03-17T20:18:48.5174698Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201844.xml 2022-03-17T20:18:49.8043468Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:49.8058244Z 2022-03-17T20:18:49.8058679Z Running tests... 2022-03-17T20:18:49.8059171Z ---------------------------------------------------------------------- 2022-03-17T20:18:51.3225028Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:51.3641288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118236 2022-03-17T20:18:51.3764411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118237 2022-03-17T20:18:52.4149223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:52.4628161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:52.4629581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:52.4654839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:52.4661724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:52.5643757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:18:54.1904878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:18:54.1905856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:18:54.4841802Z ok (4.678s) 2022-03-17T20:18:54.4842072Z 2022-03-17T20:18:54.4842465Z ---------------------------------------------------------------------- 2022-03-17T20:18:54.4843018Z Ran 1 test in 4.678s 2022-03-17T20:18:54.4843205Z 2022-03-17T20:18:54.4843299Z OK 2022-03-17T20:18:54.4843437Z 2022-03-17T20:18:54.4843575Z Generating XML reports... 2022-03-17T20:18:54.4887437Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201849.xml 2022-03-17T20:18:55.7678941Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:18:55.7693631Z 2022-03-17T20:18:55.7693888Z Running tests... 2022-03-17T20:18:55.7694294Z ---------------------------------------------------------------------- 2022-03-17T20:18:57.2822348Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:18:57.3238869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118362 2022-03-17T20:18:57.3363319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118363 2022-03-17T20:18:58.3791692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:18:58.4213132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:18:58.4213879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:58.4299792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:18:58.4306909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:18:58.5228105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:00.1607846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:00.1608861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:00.1614989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:00.1617603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:00.4441063Z ok (4.674s) 2022-03-17T20:19:00.4441301Z 2022-03-17T20:19:00.4441715Z ---------------------------------------------------------------------- 2022-03-17T20:19:00.4442054Z Ran 1 test in 4.675s 2022-03-17T20:19:00.4442189Z 2022-03-17T20:19:00.4442287Z OK 2022-03-17T20:19:00.4442400Z 2022-03-17T20:19:00.4442525Z Generating XML reports... 2022-03-17T20:19:00.4487298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201855.xml 2022-03-17T20:19:01.7415324Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:01.7431308Z 2022-03-17T20:19:01.7431753Z Running tests... 2022-03-17T20:19:01.7432255Z ---------------------------------------------------------------------- 2022-03-17T20:19:03.3202792Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:03.3630861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118488 2022-03-17T20:19:03.3757306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118489 2022-03-17T20:19:04.4202979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:04.4304705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:04.4305495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:04.4306200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:04.4313022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:04.4313779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:06.1213716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:06.1214275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:06.1221996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:06.1222461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:06.1339586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:06.1340067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:06.1348927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:06.1349406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:06.3834504Z ok (4.640s) 2022-03-17T20:19:06.3834929Z 2022-03-17T20:19:06.3835335Z ---------------------------------------------------------------------- 2022-03-17T20:19:06.3835687Z Ran 1 test in 4.640s 2022-03-17T20:19:06.3835894Z 2022-03-17T20:19:06.3835977Z OK 2022-03-17T20:19:06.3836111Z 2022-03-17T20:19:06.3836245Z Generating XML reports... 2022-03-17T20:19:06.3882781Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201901.xml 2022-03-17T20:19:07.6771105Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:07.6786962Z 2022-03-17T20:19:07.6787186Z Running tests... 2022-03-17T20:19:07.6787756Z ---------------------------------------------------------------------- 2022-03-17T20:19:09.2027956Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:09.2452127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118614 2022-03-17T20:19:09.2582038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118615 2022-03-17T20:19:10.3041118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:10.3370031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:10.3370828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:10.3446638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:10.3454234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:10.4385406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:12.0747035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:12.0747567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:12.0755321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:12.0756121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:12.0863879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:12.0864376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:12.0872309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:12.0872799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:12.3661404Z ok (4.687s) 2022-03-17T20:19:12.3661605Z 2022-03-17T20:19:12.3662228Z ---------------------------------------------------------------------- 2022-03-17T20:19:12.3662638Z Ran 1 test in 4.687s 2022-03-17T20:19:12.3662812Z 2022-03-17T20:19:12.3662846Z OK 2022-03-17T20:19:12.3662977Z 2022-03-17T20:19:12.3663111Z Generating XML reports... 2022-03-17T20:19:12.3708308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201907.xml 2022-03-17T20:19:13.6625822Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:13.6641745Z 2022-03-17T20:19:13.6642256Z Running tests... 2022-03-17T20:19:13.6642727Z ---------------------------------------------------------------------- 2022-03-17T20:19:15.1981213Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:15.2406823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118740 2022-03-17T20:19:15.2532849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118741 2022-03-17T20:19:16.2910718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:16.3351735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:16.3352572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:16.3417329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:16.3424356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:16.4369260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:18.6830644Z 2022-03-17T20:19:18.9620700Z ok (5.298s) 2022-03-17T20:19:18.9620929Z 2022-03-17T20:19:18.9621596Z ---------------------------------------------------------------------- 2022-03-17T20:19:18.9621937Z Ran 1 test in 5.298s 2022-03-17T20:19:18.9622030Z 2022-03-17T20:19:18.9622122Z OK 2022-03-17T20:19:18.9622256Z 2022-03-17T20:19:18.9622365Z Generating XML reports... 2022-03-17T20:19:18.9666658Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201913.xml 2022-03-17T20:19:20.2654606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:20.2670771Z 2022-03-17T20:19:20.2671035Z Running tests... 2022-03-17T20:19:20.2671435Z ---------------------------------------------------------------------- 2022-03-17T20:19:21.8085828Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:21.8509899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118863 2022-03-17T20:19:21.8635237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118864 2022-03-17T20:19:22.9273722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:22.9331257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:22.9332319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:22.9375425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:22.9382470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:23.0347654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:25.5725069Z ok (5.305s) 2022-03-17T20:19:25.5725425Z 2022-03-17T20:19:25.5726343Z ---------------------------------------------------------------------- 2022-03-17T20:19:25.5726972Z Ran 1 test in 5.305s 2022-03-17T20:19:25.5727271Z 2022-03-17T20:19:25.5728106Z OK 2022-03-17T20:19:25.5728364Z 2022-03-17T20:19:25.5728492Z Generating XML reports... 2022-03-17T20:19:25.5772269Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201920.xml 2022-03-17T20:19:26.8715602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:26.8731369Z 2022-03-17T20:19:26.8731638Z Running tests... 2022-03-17T20:19:26.8732080Z ---------------------------------------------------------------------- 2022-03-17T20:19:28.4060769Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:28.4482262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118986 2022-03-17T20:19:28.4607520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118987 2022-03-17T20:19:29.4995531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:29.5360863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:29.5361648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:29.5401092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:29.5408524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:29.6376497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:32.2361422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:32.2361878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:32.5704097Z ok (5.697s) 2022-03-17T20:19:32.5704595Z 2022-03-17T20:19:32.5705152Z ---------------------------------------------------------------------- 2022-03-17T20:19:32.5705492Z Ran 1 test in 5.697s 2022-03-17T20:19:32.5705684Z 2022-03-17T20:19:32.5705776Z OK 2022-03-17T20:19:32.5705895Z 2022-03-17T20:19:32.5706021Z Generating XML reports... 2022-03-17T20:19:32.5750789Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201926.xml 2022-03-17T20:19:33.8607285Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:33.8622868Z 2022-03-17T20:19:33.8623005Z Running tests... 2022-03-17T20:19:33.8623747Z ---------------------------------------------------------------------- 2022-03-17T20:19:35.3967918Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:35.4387370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119113 2022-03-17T20:19:35.4510494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119114 2022-03-17T20:19:36.4765047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:36.4872580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:36.4873316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:36.4967437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:36.4974110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:36.5887949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:38.2124298Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:19:38.5588780Z ok (4.696s) 2022-03-17T20:19:38.5589181Z 2022-03-17T20:19:38.5589984Z ---------------------------------------------------------------------- 2022-03-17T20:19:38.5590419Z Ran 1 test in 4.697s 2022-03-17T20:19:38.5590645Z 2022-03-17T20:19:38.5590727Z OK 2022-03-17T20:19:38.5590822Z 2022-03-17T20:19:38.5592928Z Generating XML reports... 2022-03-17T20:19:38.5634158Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201933.xml 2022-03-17T20:19:39.8700007Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:39.8715561Z 2022-03-17T20:19:39.8715886Z Running tests... 2022-03-17T20:19:39.8716363Z ---------------------------------------------------------------------- 2022-03-17T20:19:41.4257101Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:41.4685752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119239 2022-03-17T20:19:41.4811607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119240 2022-03-17T20:19:42.5582985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:42.5728129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:42.5728839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:42.5785734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:42.5792269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:42.6743384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:44.2801932Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:19:44.2803547Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:19:44.5891147Z ok (4.717s) 2022-03-17T20:19:44.5891434Z 2022-03-17T20:19:44.5891981Z ---------------------------------------------------------------------- 2022-03-17T20:19:44.5892363Z Ran 1 test in 4.717s 2022-03-17T20:19:44.5892508Z 2022-03-17T20:19:44.5892620Z OK 2022-03-17T20:19:44.5892754Z 2022-03-17T20:19:44.5892839Z Generating XML reports... 2022-03-17T20:19:44.5937852Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201939.xml 2022-03-17T20:19:45.8852662Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:45.8869523Z 2022-03-17T20:19:45.8869932Z Running tests... 2022-03-17T20:19:45.8870426Z ---------------------------------------------------------------------- 2022-03-17T20:19:45.8895173Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:19:45.8895480Z 2022-03-17T20:19:45.8895948Z ---------------------------------------------------------------------- 2022-03-17T20:19:45.8896221Z Ran 1 test in 0.003s 2022-03-17T20:19:45.8896384Z 2022-03-17T20:19:45.8896501Z OK (skipped=1) 2022-03-17T20:19:45.8896686Z 2022-03-17T20:19:45.8896812Z Generating XML reports... 2022-03-17T20:19:45.8941092Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201945.xml 2022-03-17T20:19:47.0351355Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:47.0367074Z 2022-03-17T20:19:47.0367214Z Running tests... 2022-03-17T20:19:47.0367854Z ---------------------------------------------------------------------- 2022-03-17T20:19:48.5766915Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:48.6189052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119400 2022-03-17T20:19:48.6314073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119401 2022-03-17T20:19:49.6746050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:49.6835425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:49.6836154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:49.6847566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:49.6854490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:49.7851269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:52.3829623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:52.3830120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:19:52.7410422Z ok (5.704s) 2022-03-17T20:19:52.7410836Z 2022-03-17T20:19:52.7411277Z ---------------------------------------------------------------------- 2022-03-17T20:19:52.7411626Z Ran 1 test in 5.704s 2022-03-17T20:19:52.7411746Z 2022-03-17T20:19:52.7411837Z OK 2022-03-17T20:19:52.7411952Z 2022-03-17T20:19:52.7412091Z Generating XML reports... 2022-03-17T20:19:52.7456536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201947.xml 2022-03-17T20:19:54.0386309Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:19:54.0401809Z 2022-03-17T20:19:54.0402357Z Running tests... 2022-03-17T20:19:54.0402763Z ---------------------------------------------------------------------- 2022-03-17T20:19:55.5876249Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:19:55.6298955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119527 2022-03-17T20:19:55.6424529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119528 2022-03-17T20:19:56.7268569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:19:56.7345350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:19:56.7346127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:56.7370061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:19:56.7376717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:19:56.8360313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:19:59.1518974Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T20:19:59.1519889Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T20:19:59.1538085Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T20:19:59.1539060Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T20:19:59.8540708Z ok (5.813s) 2022-03-17T20:19:59.8541207Z 2022-03-17T20:19:59.8541588Z ---------------------------------------------------------------------- 2022-03-17T20:19:59.8541957Z Ran 1 test in 5.814s 2022-03-17T20:19:59.8542127Z 2022-03-17T20:19:59.8542221Z OK 2022-03-17T20:19:59.8542337Z 2022-03-17T20:19:59.8542474Z Generating XML reports... 2022-03-17T20:19:59.8590397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201954.xml 2022-03-17T20:20:01.1665535Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:01.1680984Z 2022-03-17T20:20:01.1681114Z Running tests... 2022-03-17T20:20:01.1681836Z ---------------------------------------------------------------------- 2022-03-17T20:20:02.7251830Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:20:02.7675802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119684 2022-03-17T20:20:02.7805484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119685 2022-03-17T20:20:03.8267490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:03.8507880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:03.8508904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:03.8571636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:03.8578462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:03.9520748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:07.4072284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:07.4072810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:08.6926613Z ok (7.524s) 2022-03-17T20:20:08.6926871Z 2022-03-17T20:20:08.6927235Z ---------------------------------------------------------------------- 2022-03-17T20:20:08.6927574Z Ran 1 test in 7.525s 2022-03-17T20:20:08.6927741Z 2022-03-17T20:20:08.6927814Z OK 2022-03-17T20:20:08.6927947Z 2022-03-17T20:20:08.6928335Z Generating XML reports... 2022-03-17T20:20:08.6972234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202001.xml 2022-03-17T20:20:09.9966431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:09.9982623Z 2022-03-17T20:20:09.9983008Z Running tests... 2022-03-17T20:20:09.9983475Z ---------------------------------------------------------------------- 2022-03-17T20:20:11.5344229Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:20:11.5769569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119810 2022-03-17T20:20:11.5894339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119811 2022-03-17T20:20:12.6232255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:12.6371447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:12.6372224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:12.6435943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:12.6442681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:12.7386712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:15.3186072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:15.3186955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:15.6991296Z ok (5.701s) 2022-03-17T20:20:15.6991525Z 2022-03-17T20:20:15.6991911Z ---------------------------------------------------------------------- 2022-03-17T20:20:15.6992265Z Ran 1 test in 5.701s 2022-03-17T20:20:15.6992424Z 2022-03-17T20:20:15.6992464Z OK 2022-03-17T20:20:15.6992563Z 2022-03-17T20:20:15.6992690Z Generating XML reports... 2022-03-17T20:20:15.7037522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202009.xml 2022-03-17T20:20:16.9940648Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:16.9955915Z 2022-03-17T20:20:16.9956330Z Running tests... 2022-03-17T20:20:16.9956807Z ---------------------------------------------------------------------- 2022-03-17T20:20:18.5291814Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:20:18.5712421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119937 2022-03-17T20:20:18.5836510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119938 2022-03-17T20:20:19.6247030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:19.6447950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:19.6449073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:19.6450050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:19.6457171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:19.6457784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:22.2916384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:22.2916874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:22.2926305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:22.2926743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:22.7934861Z ok (5.798s) 2022-03-17T20:20:22.7935119Z 2022-03-17T20:20:22.7935505Z ---------------------------------------------------------------------- 2022-03-17T20:20:22.7935846Z Ran 1 test in 5.798s 2022-03-17T20:20:22.7935991Z 2022-03-17T20:20:22.7936067Z OK 2022-03-17T20:20:22.7936184Z 2022-03-17T20:20:22.7936313Z Generating XML reports... 2022-03-17T20:20:22.7980431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202016.xml 2022-03-17T20:20:24.0882571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:24.0897718Z 2022-03-17T20:20:24.0897967Z Running tests... 2022-03-17T20:20:24.0898461Z ---------------------------------------------------------------------- 2022-03-17T20:20:25.6101566Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:20:25.6528922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120064 2022-03-17T20:20:25.6652435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120065 2022-03-17T20:20:26.7087423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:26.7133974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:26.7135070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:26.7188813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:26.7196236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:26.7200110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:20:26.8148246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:26.8153269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:20:26.8153938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:20:26.8215970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:20:29.4377631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:29.4378184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:29.4388506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:29.4389173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:29.9751309Z ok (5.885s) 2022-03-17T20:20:29.9751537Z 2022-03-17T20:20:29.9752210Z ---------------------------------------------------------------------- 2022-03-17T20:20:29.9752526Z Ran 1 test in 5.885s 2022-03-17T20:20:29.9752695Z 2022-03-17T20:20:29.9753837Z OK 2022-03-17T20:20:29.9754760Z 2022-03-17T20:20:29.9754926Z Generating XML reports... 2022-03-17T20:20:29.9797738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202024.xml 2022-03-17T20:20:31.2861861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:31.2877703Z 2022-03-17T20:20:31.2878143Z Running tests... 2022-03-17T20:20:31.2878642Z ---------------------------------------------------------------------- 2022-03-17T20:20:32.8182835Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:20:32.8606026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120195 2022-03-17T20:20:32.8730891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120196 2022-03-17T20:20:33.9456157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:33.9728438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:33.9729248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:33.9760259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:33.9766663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:33.9769151Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-03-17T20:20:34.0743717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:34.0744947Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-03-17T20:20:36.5507421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:36.5507994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:36.5517727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:36.5518190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:36.5715583Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-03-17T20:20:36.5718523Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-03-17T20:20:36.7734346Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-03-17T20:20:36.7735543Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-03-17T20:20:36.7790746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:36.7791208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:36.7802208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:36.7802684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:37.2832851Z ok (5.995s) 2022-03-17T20:20:37.2833066Z 2022-03-17T20:20:37.2833451Z ---------------------------------------------------------------------- 2022-03-17T20:20:37.2833789Z Ran 1 test in 5.995s 2022-03-17T20:20:37.2833966Z 2022-03-17T20:20:37.2834059Z OK 2022-03-17T20:20:37.2834182Z 2022-03-17T20:20:37.2834312Z Generating XML reports... 2022-03-17T20:20:37.2879307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202031.xml 2022-03-17T20:20:38.5840726Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:38.5856138Z 2022-03-17T20:20:38.5856432Z Running tests... 2022-03-17T20:20:38.5856863Z ---------------------------------------------------------------------- 2022-03-17T20:20:40.1189767Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:20:40.1612134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120322 2022-03-17T20:20:40.1738071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120323 2022-03-17T20:20:41.2164449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:20:41.2613446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:20:41.2614473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:41.2671023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:20:41.2678192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:20:41.2681113Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:20:41.3628402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:20:41.3629558Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:20:43.9918110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:43.9918797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:43.9927859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:43.9928843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:43.9934343Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T20:20:43.9935312Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T20:20:43.9961245Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T20:20:43.9962302Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T20:20:43.9963631Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T20:20:43.9964991Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T20:20:43.9965635Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-03-17T20:20:43.9966273Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-03-17T20:20:44.2783428Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:20:44.2786893Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:20:44.2843138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:44.2843969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:44.2856143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:44.2856586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:20:44.2863446Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T20:20:44.2863959Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T20:20:44.2892403Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T20:20:44.2892964Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T20:20:44.2894787Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T20:20:44.2895414Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T20:20:44.9865058Z ok (6.401s) 2022-03-17T20:20:44.9865285Z 2022-03-17T20:20:44.9865906Z ---------------------------------------------------------------------- 2022-03-17T20:20:44.9866307Z Ran 1 test in 6.401s 2022-03-17T20:20:44.9866480Z 2022-03-17T20:20:44.9866532Z OK 2022-03-17T20:20:44.9866671Z 2022-03-17T20:20:44.9866806Z Generating XML reports... 2022-03-17T20:20:44.9911609Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202038.xml 2022-03-17T20:20:46.2805744Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:46.2821183Z 2022-03-17T20:20:46.2821665Z Running tests... 2022-03-17T20:20:46.2822264Z ---------------------------------------------------------------------- 2022-03-17T20:20:46.2844126Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:46.2844498Z 2022-03-17T20:20:46.2844749Z ---------------------------------------------------------------------- 2022-03-17T20:20:46.2845084Z Ran 1 test in 0.002s 2022-03-17T20:20:46.2845371Z 2022-03-17T20:20:46.2845543Z OK (skipped=1) 2022-03-17T20:20:46.2845701Z 2022-03-17T20:20:46.2845827Z Generating XML reports... 2022-03-17T20:20:46.2888684Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202046.xml 2022-03-17T20:20:47.4272948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:47.4287605Z 2022-03-17T20:20:47.4287919Z Running tests... 2022-03-17T20:20:47.4288538Z ---------------------------------------------------------------------- 2022-03-17T20:20:47.4311027Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:47.4311336Z 2022-03-17T20:20:47.4311624Z ---------------------------------------------------------------------- 2022-03-17T20:20:47.4312053Z Ran 1 test in 0.002s 2022-03-17T20:20:47.4312547Z 2022-03-17T20:20:47.4312639Z OK (skipped=1) 2022-03-17T20:20:47.4312856Z 2022-03-17T20:20:47.4312940Z Generating XML reports... 2022-03-17T20:20:47.4356320Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202047.xml 2022-03-17T20:20:48.5736988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:48.5752871Z 2022-03-17T20:20:48.5753161Z Running tests... 2022-03-17T20:20:48.5753605Z ---------------------------------------------------------------------- 2022-03-17T20:20:48.5778052Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:48.5778453Z 2022-03-17T20:20:48.5778794Z ---------------------------------------------------------------------- 2022-03-17T20:20:48.5779111Z Ran 1 test in 0.003s 2022-03-17T20:20:48.5779274Z 2022-03-17T20:20:48.5779397Z OK (skipped=1) 2022-03-17T20:20:48.5779551Z 2022-03-17T20:20:48.5779686Z Generating XML reports... 2022-03-17T20:20:48.5823800Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202048.xml 2022-03-17T20:20:49.7230299Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:49.7245664Z 2022-03-17T20:20:49.7245942Z Running tests... 2022-03-17T20:20:49.7246392Z ---------------------------------------------------------------------- 2022-03-17T20:20:49.7271145Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:49.7271543Z 2022-03-17T20:20:49.7271867Z ---------------------------------------------------------------------- 2022-03-17T20:20:49.7272195Z Ran 1 test in 0.003s 2022-03-17T20:20:49.7272362Z 2022-03-17T20:20:49.7272431Z OK (skipped=1) 2022-03-17T20:20:49.7272585Z 2022-03-17T20:20:49.7272711Z Generating XML reports... 2022-03-17T20:20:49.7316280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202049.xml 2022-03-17T20:20:50.8664260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:50.8679957Z 2022-03-17T20:20:50.8680336Z Running tests... 2022-03-17T20:20:50.8680843Z ---------------------------------------------------------------------- 2022-03-17T20:20:50.8705164Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:50.8705563Z 2022-03-17T20:20:50.8705891Z ---------------------------------------------------------------------- 2022-03-17T20:20:50.8706420Z Ran 1 test in 0.003s 2022-03-17T20:20:50.8706574Z 2022-03-17T20:20:50.8706646Z OK (skipped=1) 2022-03-17T20:20:50.8706803Z 2022-03-17T20:20:50.8706945Z Generating XML reports... 2022-03-17T20:20:50.8750121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202050.xml 2022-03-17T20:20:52.0144022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:52.0159702Z 2022-03-17T20:20:52.0160128Z Running tests... 2022-03-17T20:20:52.0160616Z ---------------------------------------------------------------------- 2022-03-17T20:20:52.0184366Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:52.0184748Z 2022-03-17T20:20:52.0185245Z ---------------------------------------------------------------------- 2022-03-17T20:20:52.0185609Z Ran 1 test in 0.003s 2022-03-17T20:20:52.0185776Z 2022-03-17T20:20:52.0185866Z OK (skipped=1) 2022-03-17T20:20:52.0186022Z 2022-03-17T20:20:52.0186149Z Generating XML reports... 2022-03-17T20:20:52.0229469Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202052.xml 2022-03-17T20:20:53.1640802Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:53.1656559Z 2022-03-17T20:20:53.1657389Z Running tests... 2022-03-17T20:20:53.1657877Z ---------------------------------------------------------------------- 2022-03-17T20:20:53.1681713Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:53.1682312Z 2022-03-17T20:20:53.1682879Z ---------------------------------------------------------------------- 2022-03-17T20:20:53.1683276Z Ran 1 test in 0.003s 2022-03-17T20:20:53.1683570Z 2022-03-17T20:20:53.1683702Z OK (skipped=1) 2022-03-17T20:20:53.1683865Z 2022-03-17T20:20:53.1683992Z Generating XML reports... 2022-03-17T20:20:53.1726603Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202053.xml 2022-03-17T20:20:54.3190418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:54.3206194Z 2022-03-17T20:20:54.3206567Z Running tests... 2022-03-17T20:20:54.3207198Z ---------------------------------------------------------------------- 2022-03-17T20:20:54.3231677Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:54.3232066Z 2022-03-17T20:20:54.3232481Z ---------------------------------------------------------------------- 2022-03-17T20:20:54.3232864Z Ran 1 test in 0.003s 2022-03-17T20:20:54.3233033Z 2022-03-17T20:20:54.3233144Z OK (skipped=1) 2022-03-17T20:20:54.3233306Z 2022-03-17T20:20:54.3233434Z Generating XML reports... 2022-03-17T20:20:54.3277549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202054.xml 2022-03-17T20:20:55.4669409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:55.4684867Z 2022-03-17T20:20:55.4685073Z Running tests... 2022-03-17T20:20:55.4685498Z ---------------------------------------------------------------------- 2022-03-17T20:20:55.4710008Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:55.4710383Z 2022-03-17T20:20:55.4710672Z ---------------------------------------------------------------------- 2022-03-17T20:20:55.4710991Z Ran 1 test in 0.003s 2022-03-17T20:20:55.4711180Z 2022-03-17T20:20:55.4711291Z OK (skipped=1) 2022-03-17T20:20:55.4711447Z 2022-03-17T20:20:55.4711582Z Generating XML reports... 2022-03-17T20:20:55.4755146Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202055.xml 2022-03-17T20:20:56.6162479Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:56.6178326Z 2022-03-17T20:20:56.6178397Z Running tests... 2022-03-17T20:20:56.6179477Z ---------------------------------------------------------------------- 2022-03-17T20:20:56.6203924Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:56.6204344Z 2022-03-17T20:20:56.6204623Z ---------------------------------------------------------------------- 2022-03-17T20:20:56.6204967Z Ran 1 test in 0.003s 2022-03-17T20:20:56.6205252Z 2022-03-17T20:20:56.6205411Z OK (skipped=1) 2022-03-17T20:20:56.6205554Z 2022-03-17T20:20:56.6205684Z Generating XML reports... 2022-03-17T20:20:56.6249023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202056.xml 2022-03-17T20:20:57.7699492Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:57.7715459Z 2022-03-17T20:20:57.7715723Z Running tests... 2022-03-17T20:20:57.7716117Z ---------------------------------------------------------------------- 2022-03-17T20:20:57.7738554Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:57.7738879Z 2022-03-17T20:20:57.7739209Z ---------------------------------------------------------------------- 2022-03-17T20:20:57.7739530Z Ran 1 test in 0.002s 2022-03-17T20:20:57.7739695Z 2022-03-17T20:20:57.7739808Z OK (skipped=1) 2022-03-17T20:20:57.7739927Z 2022-03-17T20:20:57.7740453Z Generating XML reports... 2022-03-17T20:20:57.7784200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202057.xml 2022-03-17T20:20:58.9261952Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:20:58.9277531Z 2022-03-17T20:20:58.9277957Z Running tests... 2022-03-17T20:20:58.9278472Z ---------------------------------------------------------------------- 2022-03-17T20:20:58.9300757Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:20:58.9301415Z 2022-03-17T20:20:58.9301828Z ---------------------------------------------------------------------- 2022-03-17T20:20:58.9302315Z Ran 1 test in 0.002s 2022-03-17T20:20:58.9302479Z 2022-03-17T20:20:58.9302567Z OK (skipped=1) 2022-03-17T20:20:58.9302724Z 2022-03-17T20:20:58.9302858Z Generating XML reports... 2022-03-17T20:20:58.9346567Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202058.xml 2022-03-17T20:21:00.0731963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:00.0747169Z 2022-03-17T20:21:00.0747599Z Running tests... 2022-03-17T20:21:00.0748135Z ---------------------------------------------------------------------- 2022-03-17T20:21:01.6574780Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:21:01.7021873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120869 2022-03-17T20:21:01.7154944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120870 2022-03-17T20:21:02.7631330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:02.7699621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:02.7700840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:02.7732175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:02.7738563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:02.8716422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:05.5165607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.5166164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.5297008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.5297476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.5408019Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:21:05.5409972Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:21:05.5732363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.5732863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.5865378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.5865875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.5954962Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:21:05.5955988Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:21:05.5957133Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:21:05.5957950Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:21:05.6002471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.6002938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.6132325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:05.6132754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:06.0257924Z ok (5.951s) 2022-03-17T20:21:06.0258130Z 2022-03-17T20:21:06.0258516Z ---------------------------------------------------------------------- 2022-03-17T20:21:06.0258870Z Ran 1 test in 5.951s 2022-03-17T20:21:06.0259035Z 2022-03-17T20:21:06.0259125Z OK 2022-03-17T20:21:06.0259250Z 2022-03-17T20:21:06.0259385Z Generating XML reports... 2022-03-17T20:21:06.0303366Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202100.xml 2022-03-17T20:21:07.3395006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:07.3409993Z 2022-03-17T20:21:07.3410398Z Running tests... 2022-03-17T20:21:07.3410871Z ---------------------------------------------------------------------- 2022-03-17T20:21:08.8764881Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:21:08.9184286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120996 2022-03-17T20:21:08.9309026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120997 2022-03-17T20:21:10.0135234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:10.0322903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:10.0323691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:10.0338230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:10.0345318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:10.1337100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:12.5395805Z ok (5.198s) 2022-03-17T20:21:12.5396018Z 2022-03-17T20:21:12.5396668Z ---------------------------------------------------------------------- 2022-03-17T20:21:12.5396978Z Ran 1 test in 5.199s 2022-03-17T20:21:12.5397125Z 2022-03-17T20:21:12.5397226Z OK 2022-03-17T20:21:12.5397386Z 2022-03-17T20:21:12.5397478Z Generating XML reports... 2022-03-17T20:21:12.5442293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202107.xml 2022-03-17T20:21:13.8291452Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:13.8306700Z 2022-03-17T20:21:13.8306894Z Running tests... 2022-03-17T20:21:13.8307304Z ---------------------------------------------------------------------- 2022-03-17T20:21:15.3607628Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:21:15.4026704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121118 2022-03-17T20:21:15.4151234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121119 2022-03-17T20:21:16.4850744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:16.5034329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:16.5035093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:16.5053363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:16.5060531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:16.6050199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:19.1592279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:19.1592893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:19.5245697Z ok (5.694s) 2022-03-17T20:21:19.5245912Z 2022-03-17T20:21:19.5246509Z ---------------------------------------------------------------------- 2022-03-17T20:21:19.5246892Z Ran 1 test in 5.694s 2022-03-17T20:21:19.5247054Z 2022-03-17T20:21:19.5247075Z OK 2022-03-17T20:21:19.5247203Z 2022-03-17T20:21:19.5247333Z Generating XML reports... 2022-03-17T20:21:19.5291441Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202113.xml 2022-03-17T20:21:20.8206304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:20.8222131Z 2022-03-17T20:21:20.8222232Z Running tests... 2022-03-17T20:21:20.8223096Z ---------------------------------------------------------------------- 2022-03-17T20:21:20.8300956Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip (0.008s) 2022-03-17T20:21:20.8301635Z 2022-03-17T20:21:20.8301897Z ---------------------------------------------------------------------- 2022-03-17T20:21:20.8302205Z Ran 1 test in 0.008s 2022-03-17T20:21:20.8302359Z 2022-03-17T20:21:20.8302468Z OK (skipped=1) 2022-03-17T20:21:20.8303375Z 2022-03-17T20:21:20.8303779Z Generating XML reports... 2022-03-17T20:21:20.8346775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202120.xml 2022-03-17T20:21:21.9295583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:21.9311196Z 2022-03-17T20:21:21.9311334Z Running tests... 2022-03-17T20:21:21.9312233Z ---------------------------------------------------------------------- 2022-03-17T20:21:23.4803142Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:21:23.5223313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121280 2022-03-17T20:21:23.5347612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121281 2022-03-17T20:21:24.5764677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:24.6246224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:24.6246958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:24.6270797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:24.6277300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:24.7261331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:27.3381754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:27.3382279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:21:27.7444257Z ok (5.813s) 2022-03-17T20:21:27.7444595Z 2022-03-17T20:21:27.7445351Z ---------------------------------------------------------------------- 2022-03-17T20:21:27.7445784Z Ran 1 test in 5.813s 2022-03-17T20:21:27.7445948Z 2022-03-17T20:21:27.7446055Z OK 2022-03-17T20:21:27.7446198Z 2022-03-17T20:21:27.7446289Z Generating XML reports... 2022-03-17T20:21:27.7491190Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202121.xml 2022-03-17T20:21:29.0534051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:29.0549907Z 2022-03-17T20:21:29.0550313Z Running tests... 2022-03-17T20:21:29.0551155Z ---------------------------------------------------------------------- 2022-03-17T20:21:30.6075788Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:21:30.6499632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121407 2022-03-17T20:21:30.6626337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121408 2022-03-17T20:21:31.7391886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:31.7597063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:31.7597836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:31.7696133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:31.7703484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:31.8609481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:31.8720530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:21:31.8721072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:21:31.8722432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:21:31.8723273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:21:31.8724436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:21:31.8725100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:21:31.8725912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:21:31.8726599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:21:33.5701972Z ok (4.515s) 2022-03-17T20:21:33.5702394Z 2022-03-17T20:21:33.5702796Z ---------------------------------------------------------------------- 2022-03-17T20:21:33.5703136Z Ran 1 test in 4.515s 2022-03-17T20:21:33.5703304Z 2022-03-17T20:21:33.5703395Z OK 2022-03-17T20:21:33.5703534Z 2022-03-17T20:21:33.5704397Z Generating XML reports... 2022-03-17T20:21:33.5749557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202129.xml 2022-03-17T20:21:34.8749317Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:34.8764504Z 2022-03-17T20:21:34.8764708Z Running tests... 2022-03-17T20:21:34.8765127Z ---------------------------------------------------------------------- 2022-03-17T20:21:36.4128937Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:21:36.4553173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121537 2022-03-17T20:21:36.4677248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121538 2022-03-17T20:21:37.5383680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:37.5564385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:37.5565160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:37.5586538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:37.5593528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:37.6576690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:37.6687816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:21:37.6688274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:21:37.6688936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:21:37.6689630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:21:37.6692308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:21:37.6794431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:21:37.6795065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:21:37.6795737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:21:50.1952198Z ok (15.318s) 2022-03-17T20:21:50.1952749Z 2022-03-17T20:21:50.1953160Z ---------------------------------------------------------------------- 2022-03-17T20:21:50.1953491Z Ran 1 test in 15.319s 2022-03-17T20:21:50.1953667Z 2022-03-17T20:21:50.1953763Z OK 2022-03-17T20:21:50.1953915Z 2022-03-17T20:21:50.1954046Z Generating XML reports... 2022-03-17T20:21:50.2001136Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202134.xml 2022-03-17T20:21:51.4977198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:51.4992598Z 2022-03-17T20:21:51.4992995Z Running tests... 2022-03-17T20:21:51.4993682Z ---------------------------------------------------------------------- 2022-03-17T20:21:53.0500494Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:21:53.0923680Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121667 2022-03-17T20:21:53.1048758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121668 2022-03-17T20:21:54.1721491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:21:54.1901725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:21:54.1902682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:54.1924396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:21:54.1930706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:21:54.2917859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:21:57.7156031Z ok (6.216s) 2022-03-17T20:21:57.7156229Z 2022-03-17T20:21:57.7156614Z ---------------------------------------------------------------------- 2022-03-17T20:21:57.7156971Z Ran 1 test in 6.216s 2022-03-17T20:21:57.7157138Z 2022-03-17T20:21:57.7157231Z OK 2022-03-17T20:21:57.7157347Z 2022-03-17T20:21:57.7157488Z Generating XML reports... 2022-03-17T20:21:57.7202894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202151.xml 2022-03-17T20:21:59.0122630Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:21:59.0138488Z 2022-03-17T20:21:59.0138584Z Running tests... 2022-03-17T20:21:59.0139374Z ---------------------------------------------------------------------- 2022-03-17T20:22:00.5590327Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:00.6022641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121794 2022-03-17T20:22:00.6151487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121795 2022-03-17T20:22:01.6869105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:01.7042689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:01.7043472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:01.7071580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:01.7078365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:01.8058669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:05.2258365Z ok (6.212s) 2022-03-17T20:22:05.2258790Z 2022-03-17T20:22:05.2259510Z ---------------------------------------------------------------------- 2022-03-17T20:22:05.2260463Z Ran 1 test in 6.212s 2022-03-17T20:22:05.2260658Z 2022-03-17T20:22:05.2260763Z OK 2022-03-17T20:22:05.2260894Z 2022-03-17T20:22:05.2261013Z Generating XML reports... 2022-03-17T20:22:05.2306221Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202159.xml 2022-03-17T20:22:06.5209159Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:06.5223944Z 2022-03-17T20:22:06.5224151Z Running tests... 2022-03-17T20:22:06.5225028Z ---------------------------------------------------------------------- 2022-03-17T20:22:08.0675075Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:08.1099967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121921 2022-03-17T20:22:08.1225541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121922 2022-03-17T20:22:09.1697554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:09.1716325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:09.1717334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:09.1798608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:09.1805482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:09.2732239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:12.1319299Z ok (5.609s) 2022-03-17T20:22:12.1319534Z 2022-03-17T20:22:12.1319900Z ---------------------------------------------------------------------- 2022-03-17T20:22:12.1320249Z Ran 1 test in 5.609s 2022-03-17T20:22:12.1320428Z 2022-03-17T20:22:12.1320522Z OK 2022-03-17T20:22:12.1320666Z 2022-03-17T20:22:12.1320780Z Generating XML reports... 2022-03-17T20:22:12.1364613Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202206.xml 2022-03-17T20:22:13.4329242Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:13.4345447Z 2022-03-17T20:22:13.4345633Z Running tests... 2022-03-17T20:22:13.4346161Z ---------------------------------------------------------------------- 2022-03-17T20:22:14.9654479Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:15.0083273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122044 2022-03-17T20:22:15.0211258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122045 2022-03-17T20:22:16.0886321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:16.1366367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:16.1367169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:16.1392915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:16.1399574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:16.2381293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:18.8154329Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:18.8268444Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:18.8330945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:18.8331453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:19.1304600Z ok (5.695s) 2022-03-17T20:22:19.1304854Z 2022-03-17T20:22:19.1305251Z ---------------------------------------------------------------------- 2022-03-17T20:22:19.1305568Z Ran 1 test in 5.696s 2022-03-17T20:22:19.1305737Z 2022-03-17T20:22:19.1305879Z OK 2022-03-17T20:22:19.1305974Z 2022-03-17T20:22:19.1306107Z Generating XML reports... 2022-03-17T20:22:19.1349880Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202213.xml 2022-03-17T20:22:20.4375467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:20.4391102Z 2022-03-17T20:22:20.4391382Z Running tests... 2022-03-17T20:22:20.4391833Z ---------------------------------------------------------------------- 2022-03-17T20:22:21.9680600Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:22.0101977Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122171 2022-03-17T20:22:22.0227610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122172 2022-03-17T20:22:23.0764951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:23.0967744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:23.0968274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:23.0968966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:23.0975555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:23.0976021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:25.4675589Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:22:25.4676496Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:22:25.4677729Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:22:25.4678551Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:22:26.1336600Z ok (5.694s) 2022-03-17T20:22:26.1336747Z 2022-03-17T20:22:26.1337160Z ---------------------------------------------------------------------- 2022-03-17T20:22:26.1337501Z Ran 1 test in 5.695s 2022-03-17T20:22:26.1337620Z 2022-03-17T20:22:26.1337715Z OK 2022-03-17T20:22:26.1337850Z 2022-03-17T20:22:26.1337985Z Generating XML reports... 2022-03-17T20:22:26.1382612Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202220.xml 2022-03-17T20:22:27.4315835Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:27.4331179Z 2022-03-17T20:22:27.4331263Z Running tests... 2022-03-17T20:22:27.4332484Z ---------------------------------------------------------------------- 2022-03-17T20:22:28.9754673Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:29.0179307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122298 2022-03-17T20:22:29.0306515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122299 2022-03-17T20:22:30.0713770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:30.0764827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:30.0765969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:30.0815540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:30.0822806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:30.1779894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:32.2322490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:32.2323855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:32.2628261Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:32.2629784Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:32.6410243Z ok (5.208s) 2022-03-17T20:22:32.6410404Z 2022-03-17T20:22:32.6410911Z ---------------------------------------------------------------------- 2022-03-17T20:22:32.6411434Z Ran 1 test in 5.208s 2022-03-17T20:22:32.6411615Z 2022-03-17T20:22:32.6411654Z OK 2022-03-17T20:22:32.6411793Z 2022-03-17T20:22:32.6411903Z Generating XML reports... 2022-03-17T20:22:32.6458394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202227.xml 2022-03-17T20:22:33.9512816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:33.9527998Z 2022-03-17T20:22:33.9528378Z Running tests... 2022-03-17T20:22:33.9528881Z ---------------------------------------------------------------------- 2022-03-17T20:22:35.4984070Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:35.5407658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122428 2022-03-17T20:22:35.5533462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122429 2022-03-17T20:22:36.6486812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:36.6520739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:36.6521747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:36.6588266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:36.6594653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:36.7535366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:38.7981996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:38.7982569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:38.8316121Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:38.8317651Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:39.1621746Z ok (5.209s) 2022-03-17T20:22:39.1622170Z 2022-03-17T20:22:39.1622833Z ---------------------------------------------------------------------- 2022-03-17T20:22:39.1623370Z Ran 1 test in 5.209s 2022-03-17T20:22:39.1623697Z 2022-03-17T20:22:39.1623889Z OK 2022-03-17T20:22:39.1624189Z 2022-03-17T20:22:39.1624352Z Generating XML reports... 2022-03-17T20:22:39.1669941Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202233.xml 2022-03-17T20:22:40.4602363Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:40.4617201Z 2022-03-17T20:22:40.4617404Z Running tests... 2022-03-17T20:22:40.4617850Z ---------------------------------------------------------------------- 2022-03-17T20:22:41.9812155Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:42.0237305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122558 2022-03-17T20:22:42.0363720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122559 2022-03-17T20:22:43.0864523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:43.1033983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:43.1034756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:43.1067298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:43.1073964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:43.2048376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:45.8453723Z ok (5.383s) 2022-03-17T20:22:45.8455520Z 2022-03-17T20:22:45.8456206Z ---------------------------------------------------------------------- 2022-03-17T20:22:45.8456575Z Ran 1 test in 5.384s 2022-03-17T20:22:45.8456741Z 2022-03-17T20:22:45.8456829Z OK 2022-03-17T20:22:45.8457210Z 2022-03-17T20:22:45.8457359Z Generating XML reports... 2022-03-17T20:22:45.8499874Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202240.xml 2022-03-17T20:22:47.1419925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:47.1435547Z 2022-03-17T20:22:47.1435863Z Running tests... 2022-03-17T20:22:47.1436343Z ---------------------------------------------------------------------- 2022-03-17T20:22:48.6817044Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:48.7241118Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122681 2022-03-17T20:22:48.7365223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122682 2022-03-17T20:22:49.7758700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:49.7944463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:49.7945244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:49.7961420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:49.7968163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:49.8958898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:51.2061748Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:22:51.2062611Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:22:51.2063737Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:22:51.2064562Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:22:51.4934897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:51.4935434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:51.4985857Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:51.4988110Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:22:51.5090481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:51.5090982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:51.5150904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:51.5151390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:51.8446284Z ok (4.701s) 2022-03-17T20:22:51.8446601Z 2022-03-17T20:22:51.8447158Z ---------------------------------------------------------------------- 2022-03-17T20:22:51.8447472Z Ran 1 test in 4.701s 2022-03-17T20:22:51.8447654Z 2022-03-17T20:22:51.8447707Z OK 2022-03-17T20:22:51.8447956Z 2022-03-17T20:22:51.8448191Z Generating XML reports... 2022-03-17T20:22:51.8496026Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202247.xml 2022-03-17T20:22:53.1546717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:53.1562077Z 2022-03-17T20:22:53.1562382Z Running tests... 2022-03-17T20:22:53.1562826Z ---------------------------------------------------------------------- 2022-03-17T20:22:54.7127349Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:22:54.7551861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122807 2022-03-17T20:22:54.7679088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122808 2022-03-17T20:22:55.8217404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:22:55.8378794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:22:55.8379596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:55.8419960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:22:55.8426993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:22:55.9393539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:22:57.2578815Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:22:57.2579741Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:22:57.2581087Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:22:57.2582227Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:22:57.5456827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:57.5457380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:22:57.8757190Z ok (4.719s) 2022-03-17T20:22:57.8757384Z 2022-03-17T20:22:57.8757776Z ---------------------------------------------------------------------- 2022-03-17T20:22:57.8758146Z Ran 1 test in 4.720s 2022-03-17T20:22:57.8758292Z 2022-03-17T20:22:57.8758370Z OK 2022-03-17T20:22:57.8758503Z 2022-03-17T20:22:57.8758639Z Generating XML reports... 2022-03-17T20:22:57.8805623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202253.xml 2022-03-17T20:22:59.1786281Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:22:59.1801296Z 2022-03-17T20:22:59.1801592Z Running tests... 2022-03-17T20:22:59.1802041Z ---------------------------------------------------------------------- 2022-03-17T20:23:00.7389472Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:23:00.7815121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122933 2022-03-17T20:23:00.7941758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122934 2022-03-17T20:23:01.8825491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:01.8925280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:01.8926487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:01.8927194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:01.8935587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:01.8936157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:03.5745041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:03.5749127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:03.5828835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:03.5832089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:03.9022106Z ok (4.722s) 2022-03-17T20:23:03.9022370Z 2022-03-17T20:23:03.9022758Z ---------------------------------------------------------------------- 2022-03-17T20:23:03.9023098Z Ran 1 test in 4.722s 2022-03-17T20:23:03.9023244Z 2022-03-17T20:23:03.9023363Z OK 2022-03-17T20:23:03.9023500Z 2022-03-17T20:23:03.9023633Z Generating XML reports... 2022-03-17T20:23:03.9069011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202259.xml 2022-03-17T20:23:05.2030262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:05.2046576Z 2022-03-17T20:23:05.2046775Z Running tests... 2022-03-17T20:23:05.2047242Z ---------------------------------------------------------------------- 2022-03-17T20:23:06.7517627Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:23:06.7943084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123059 2022-03-17T20:23:06.8069039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123060 2022-03-17T20:23:07.8808304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:07.8854897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:07.8855752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:07.8909544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:07.8916850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:07.9869695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:09.6717726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:09.6719058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:10.6161432Z ok (5.411s) 2022-03-17T20:23:10.6161897Z 2022-03-17T20:23:10.6162367Z ---------------------------------------------------------------------- 2022-03-17T20:23:10.6162698Z Ran 1 test in 5.411s 2022-03-17T20:23:10.6162869Z 2022-03-17T20:23:10.6162945Z OK 2022-03-17T20:23:10.6163083Z 2022-03-17T20:23:10.6163220Z Generating XML reports... 2022-03-17T20:23:10.6208981Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202305.xml 2022-03-17T20:23:11.9004100Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:11.9018761Z 2022-03-17T20:23:11.9019091Z Running tests... 2022-03-17T20:23:11.9019503Z ---------------------------------------------------------------------- 2022-03-17T20:23:13.4217303Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:23:13.4638119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123189 2022-03-17T20:23:13.4764636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123190 2022-03-17T20:23:14.4701935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:14.5066350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:14.5067540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:14.5107084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:14.5114508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:14.6081945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:17.2868424Z ok (5.385s) 2022-03-17T20:23:17.2868664Z 2022-03-17T20:23:17.2869043Z ---------------------------------------------------------------------- 2022-03-17T20:23:17.2869414Z Ran 1 test in 5.385s 2022-03-17T20:23:17.2869583Z 2022-03-17T20:23:17.2869680Z OK 2022-03-17T20:23:17.2869803Z 2022-03-17T20:23:17.2869952Z Generating XML reports... 2022-03-17T20:23:17.2915995Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202311.xml 2022-03-17T20:23:18.5935063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:18.5950576Z 2022-03-17T20:23:18.5950718Z Running tests... 2022-03-17T20:23:18.5951159Z ---------------------------------------------------------------------- 2022-03-17T20:23:20.1233473Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:23:20.1658207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123312 2022-03-17T20:23:20.1782591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123313 2022-03-17T20:23:21.2238963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:21.2561275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:21.2562059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:21.2644170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:21.2651330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:21.3575678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:23.8873016Z ok (5.292s) 2022-03-17T20:23:23.8873231Z 2022-03-17T20:23:23.8873630Z ---------------------------------------------------------------------- 2022-03-17T20:23:23.8873952Z Ran 1 test in 5.292s 2022-03-17T20:23:23.8874138Z 2022-03-17T20:23:23.8874237Z OK 2022-03-17T20:23:23.8874372Z 2022-03-17T20:23:23.8877394Z Generating XML reports... 2022-03-17T20:23:23.8918732Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202318.xml 2022-03-17T20:23:25.1848113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:25.1863606Z 2022-03-17T20:23:25.1864060Z Running tests... 2022-03-17T20:23:25.1864543Z ---------------------------------------------------------------------- 2022-03-17T20:23:26.7218882Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:23:26.7642592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123435 2022-03-17T20:23:26.7769449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123436 2022-03-17T20:23:27.8792273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:27.8819915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:27.8821326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:27.8893336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:27.8899878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:27.9835516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:30.5770041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:30.5770606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:30.8868746Z ok (5.700s) 2022-03-17T20:23:30.8868943Z 2022-03-17T20:23:30.8869321Z ---------------------------------------------------------------------- 2022-03-17T20:23:30.8869651Z Ran 1 test in 5.701s 2022-03-17T20:23:30.8869819Z 2022-03-17T20:23:30.8869912Z OK 2022-03-17T20:23:30.8870047Z 2022-03-17T20:23:30.8870176Z Generating XML reports... 2022-03-17T20:23:30.8914772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202325.xml 2022-03-17T20:23:32.1805986Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:23:32.1821118Z 2022-03-17T20:23:32.1821551Z Running tests... 2022-03-17T20:23:32.1822216Z ---------------------------------------------------------------------- 2022-03-17T20:23:33.7261358Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:23:33.7680511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123562 2022-03-17T20:23:33.7810207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123563 2022-03-17T20:23:34.7709998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:23:34.8198477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:23:34.8199299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:34.8216225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:23:34.8223075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:23:34.9212843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:23:36.1196595Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:23:36.2083948Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:23:36.2086869Z Running test: batch_norm_net sync interval 2022-03-17T20:23:36.2087353Z 1 with iteration mapping 2022-03-17T20:23:36.2087821Z {0: 0, 1: 2} 2022-03-17T20:23:36.9074171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:36.9108366Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:36.9109216Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:36.9110225Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:36.9110998Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:36.9191318Z Running test: batch_norm_net sync interval 2022-03-17T20:23:36.9191653Z 1 with iteration mapping 2022-03-17T20:23:36.9191914Z {0: 0, 1: 3} 2022-03-17T20:23:36.9230971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:36.9279596Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:36.9280377Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:36.9281407Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:36.9282155Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:36.9347937Z Running test: batch_norm_net sync interval 2022-03-17T20:23:36.9348276Z 1 with iteration mapping 2022-03-17T20:23:36.9348544Z {0: 0, 1: 10} 2022-03-17T20:23:36.9387641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:36.9572870Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:36.9573806Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:36.9574866Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:36.9575638Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:36.9643248Z Running test: batch_norm_net sync interval 2022-03-17T20:23:36.9643589Z 1 with iteration mapping 2022-03-17T20:23:36.9643870Z {0: 5, 1: 7} 2022-03-17T20:23:37.5968884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:37.5969423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:37.6098108Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:37.6098918Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:37.6099944Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:37.6100971Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:37.6167442Z Running test: batch_norm_net sync interval 2022-03-17T20:23:37.6167756Z 1 with iteration mapping 2022-03-17T20:23:37.6167998Z {0: 5, 1: 8} 2022-03-17T20:23:37.6208161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:37.6208677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:37.6355101Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:37.6355947Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:37.6357203Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:37.6357953Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:37.6423185Z Running test: batch_norm_net sync interval 2022-03-17T20:23:37.6423629Z 1 with iteration mapping 2022-03-17T20:23:37.6424049Z {0: 5, 1: 15} 2022-03-17T20:23:37.6462868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:37.6463349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:37.6745325Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:37.6746148Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:37.6747186Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:37.6747937Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:37.6812368Z Running test: large_conv_model sync interval 2022-03-17T20:23:37.6812672Z 1 with iteration mapping 2022-03-17T20:23:37.6812952Z {0: 0, 1: 2} 2022-03-17T20:23:39.2322322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:40.3242164Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:40.3242934Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:40.3243948Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:40.3244713Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:40.3313541Z Running test: large_conv_model sync interval 2022-03-17T20:23:40.3313920Z 1 with iteration mapping 2022-03-17T20:23:40.3314184Z {0: 0, 1: 3} 2022-03-17T20:23:41.4242297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:43.6025019Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:43.6025846Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:43.6026867Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:43.6027653Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:43.6091092Z Running test: large_conv_model sync interval 2022-03-17T20:23:43.6091411Z 1 with iteration mapping 2022-03-17T20:23:43.6091688Z {0: 0, 1: 10} 2022-03-17T20:23:44.6901079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:54.4732068Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:54.4733215Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:54.4734291Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:23:54.4735068Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:23:54.4797933Z Running test: large_conv_model sync interval 2022-03-17T20:23:54.4798248Z 1 with iteration mapping 2022-03-17T20:23:54.4798526Z {0: 5, 1: 7} 2022-03-17T20:23:55.9701492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:23:55.9702018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:02.4934268Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:02.4935117Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:02.4936156Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:02.4936911Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:02.5007277Z Running test: large_conv_model sync interval 2022-03-17T20:24:02.5007629Z 1 with iteration mapping 2022-03-17T20:24:02.5007884Z {0: 5, 1: 8} 2022-03-17T20:24:03.5907285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:03.5907839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:11.2516403Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:11.2517207Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:11.2518262Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:11.2519031Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:11.2583279Z Running test: large_conv_model sync interval 2022-03-17T20:24:11.2583623Z 1 with iteration mapping 2022-03-17T20:24:11.2583905Z {0: 5, 1: 15} 2022-03-17T20:24:12.3480617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:12.3481141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.5908288Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.5909179Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.5910204Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.5910997Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.5974984Z Running test: small_model sync interval 2022-03-17T20:24:27.5975354Z 1 with iteration mapping 2022-03-17T20:24:27.5975578Z {0: 0, 1: 2} 2022-03-17T20:24:27.6024338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.6108486Z Running test: small_model sync interval 2022-03-17T20:24:27.6108815Z 1 with iteration mapping 2022-03-17T20:24:27.6109396Z {0: 0, 1: 3} 2022-03-17T20:24:27.6145927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.6248833Z Running test: small_model sync interval 2022-03-17T20:24:27.6249207Z 1 with iteration mapping 2022-03-17T20:24:27.6249481Z {0: 0, 1: 10} 2022-03-17T20:24:27.6286082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.6517201Z Running test: small_model sync interval 2022-03-17T20:24:27.6517651Z 1 with iteration mapping 2022-03-17T20:24:27.6517926Z {0: 5, 1: 7} 2022-03-17T20:24:27.6555690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.6556172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.6736225Z Running test: small_model sync interval 2022-03-17T20:24:27.6736799Z 1 with iteration mapping 2022-03-17T20:24:27.6737071Z {0: 5, 1: 8} 2022-03-17T20:24:27.6775181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.6775710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.6975213Z Running test: small_model sync interval 2022-03-17T20:24:27.6975635Z 1 with iteration mapping 2022-03-17T20:24:27.6975906Z {0: 5, 1: 15} 2022-03-17T20:24:27.7013661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.7014143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.7338885Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.7339366Z 1 with iteration mapping 2022-03-17T20:24:27.7339637Z {0: 0, 1: 2} 2022-03-17T20:24:27.7388489Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7389427Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7390467Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7391316Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7410309Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.7410799Z 1 with iteration mapping 2022-03-17T20:24:27.7411069Z {0: 0, 1: 3} 2022-03-17T20:24:27.7468608Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7469310Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7470335Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7471084Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7490438Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.7490983Z 1 with iteration mapping 2022-03-17T20:24:27.7491230Z {0: 0, 1: 10} 2022-03-17T20:24:27.7629866Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7630630Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7631641Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7632411Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7651905Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.7652377Z 1 with iteration mapping 2022-03-17T20:24:27.7652630Z {0: 5, 1: 7} 2022-03-17T20:24:27.7672205Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:24:27.7763362Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7764310Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7765433Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7766290Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7785985Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.7786365Z 1 with iteration mapping 2022-03-17T20:24:27.7786617Z {0: 5, 1: 8} 2022-03-17T20:24:27.7906650Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7907442Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7908482Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.7909251Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.7928623Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.7929042Z 1 with iteration mapping 2022-03-17T20:24:27.7929249Z {0: 5, 1: 15} 2022-03-17T20:24:27.8136865Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8137664Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8138704Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8139466Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8159481Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.8159851Z 1 with iteration mapping 2022-03-17T20:24:27.8160103Z {0: 0, 1: 2} 2022-03-17T20:24:27.8179861Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:24:27.8210288Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8211050Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8212203Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8212993Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8232619Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.8233016Z 1 with iteration mapping 2022-03-17T20:24:27.8233246Z {0: 0, 1: 3} 2022-03-17T20:24:27.8292455Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8293236Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8294269Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8295040Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8314885Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.8315258Z 1 with iteration mapping 2022-03-17T20:24:27.8315511Z {0: 0, 1: 10} 2022-03-17T20:24:27.8457912Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8458711Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8459754Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8460771Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8480261Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.8480675Z 1 with iteration mapping 2022-03-17T20:24:27.8480888Z {0: 5, 1: 7} 2022-03-17T20:24:27.8587784Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8588558Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8589743Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8590508Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8610065Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.8610399Z 1 with iteration mapping 2022-03-17T20:24:27.8610685Z {0: 5, 1: 8} 2022-03-17T20:24:27.8730550Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8731330Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8732371Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8733121Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8752812Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:24:27.8753218Z 1 with iteration mapping 2022-03-17T20:24:27.8753501Z {0: 5, 1: 15} 2022-03-17T20:24:27.8957189Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8958016Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8959043Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:27.8959802Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:27.8979351Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:24:27.8979721Z 1 with iteration mapping 2022-03-17T20:24:27.8979972Z {0: 0, 1: 2} 2022-03-17T20:24:27.8990227Z registered hook 2022-03-17T20:24:27.8990589Z registered hook 2022-03-17T20:24:27.9017904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.9103529Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:24:27.9103888Z 1 with iteration mapping 2022-03-17T20:24:27.9104159Z {0: 0, 1: 3} 2022-03-17T20:24:27.9113176Z registered hook 2022-03-17T20:24:27.9113525Z registered hook 2022-03-17T20:24:27.9140992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.9245708Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:24:27.9246134Z 1 with iteration mapping 2022-03-17T20:24:27.9246541Z {0: 0, 1: 10} 2022-03-17T20:24:27.9255556Z registered hook 2022-03-17T20:24:27.9255933Z registered hook 2022-03-17T20:24:27.9283683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.9522035Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:24:27.9522472Z 1 with iteration mapping 2022-03-17T20:24:27.9522701Z {0: 5, 1: 7} 2022-03-17T20:24:27.9532050Z registered hook 2022-03-17T20:24:27.9532576Z registered hook 2022-03-17T20:24:27.9561225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.9561715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.9748375Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:24:27.9748820Z 1 with iteration mapping 2022-03-17T20:24:27.9749053Z {0: 5, 1: 8} 2022-03-17T20:24:27.9758377Z registered hook 2022-03-17T20:24:27.9758755Z registered hook 2022-03-17T20:24:27.9787965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.9788448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:27.9993780Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:24:27.9994210Z 1 with iteration mapping 2022-03-17T20:24:27.9994450Z {0: 5, 1: 15} 2022-03-17T20:24:28.0003700Z registered hook 2022-03-17T20:24:28.0004046Z registered hook 2022-03-17T20:24:28.0033040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.0033531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.0369611Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:24:28.0370010Z 1 with iteration mapping 2022-03-17T20:24:28.0370271Z {0: 0, 1: 2} 2022-03-17T20:24:28.0379552Z registered hook 2022-03-17T20:24:28.0379931Z registered hook 2022-03-17T20:24:28.0388378Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T20:24:28.0393042Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T20:24:28.0409481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.0413343Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T20:24:28.0417886Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T20:24:28.0505134Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:24:28.0505528Z 1 with iteration mapping 2022-03-17T20:24:28.0505770Z {0: 0, 1: 3} 2022-03-17T20:24:28.0514868Z registered hook 2022-03-17T20:24:28.0515287Z registered hook 2022-03-17T20:24:28.0547576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.0660812Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:24:28.0661391Z 1 with iteration mapping 2022-03-17T20:24:28.0661840Z {0: 0, 1: 10} 2022-03-17T20:24:28.0671962Z registered hook 2022-03-17T20:24:28.0672345Z registered hook 2022-03-17T20:24:28.0702991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.0981789Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:24:28.0982242Z 1 with iteration mapping 2022-03-17T20:24:28.0982431Z {0: 5, 1: 7} 2022-03-17T20:24:28.0991836Z registered hook 2022-03-17T20:24:28.0992352Z registered hook 2022-03-17T20:24:28.1027656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.1028172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.1244987Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:24:28.1245438Z 1 with iteration mapping 2022-03-17T20:24:28.1245645Z {0: 5, 1: 8} 2022-03-17T20:24:28.1254886Z registered hook 2022-03-17T20:24:28.1255259Z registered hook 2022-03-17T20:24:28.1289983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.1290536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.1530268Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:24:28.1530673Z 1 with iteration mapping 2022-03-17T20:24:28.1530884Z {0: 5, 1: 15} 2022-03-17T20:24:28.1540025Z registered hook 2022-03-17T20:24:28.1540365Z registered hook 2022-03-17T20:24:28.1575911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.1576419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.1943868Z Running test: batch_norm_net sync interval 2022-03-17T20:24:28.1944278Z 2 with iteration mapping 2022-03-17T20:24:28.1944711Z {0: 0, 1: 2} 2022-03-17T20:24:28.1984335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.2011157Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2011953Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2012997Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2013744Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2080149Z Running test: batch_norm_net sync interval 2022-03-17T20:24:28.2080569Z 2 with iteration mapping 2022-03-17T20:24:28.2081046Z {0: 0, 1: 3} 2022-03-17T20:24:28.2119729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.2162054Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2162996Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2164003Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2164843Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2232759Z Running test: batch_norm_net sync interval 2022-03-17T20:24:28.2233262Z 2 with iteration mapping 2022-03-17T20:24:28.2233600Z {0: 0, 1: 10} 2022-03-17T20:24:28.2272957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.2429579Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2430405Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2431420Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2432169Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2498351Z Running test: batch_norm_net sync interval 2022-03-17T20:24:28.2498755Z 2 with iteration mapping 2022-03-17T20:24:28.2499196Z {0: 5, 1: 7} 2022-03-17T20:24:28.2539159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.2539845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.2647064Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2647847Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2648863Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2649605Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2716074Z Running test: batch_norm_net sync interval 2022-03-17T20:24:28.2716407Z 2 with iteration mapping 2022-03-17T20:24:28.2716838Z {0: 5, 1: 8} 2022-03-17T20:24:28.2755963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.2756658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.2882725Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2883660Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2884692Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.2885510Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.2952114Z Running test: batch_norm_net sync interval 2022-03-17T20:24:28.2952508Z 2 with iteration mapping 2022-03-17T20:24:28.2952953Z {0: 5, 1: 15} 2022-03-17T20:24:28.2992195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.2992850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:28.3230503Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.3231287Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.3232311Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:28.3233063Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:28.3299374Z Running test: large_conv_model sync interval 2022-03-17T20:24:28.3299726Z 3 with iteration mapping 2022-03-17T20:24:28.3300186Z {0: 0, 1: 2} 2022-03-17T20:24:29.4249918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:30.5092109Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:30.5092893Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:30.5093949Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:30.5094698Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:30.5165751Z Running test: large_conv_model sync interval 2022-03-17T20:24:30.5166077Z 3 with iteration mapping 2022-03-17T20:24:30.5166359Z {0: 0, 1: 3} 2022-03-17T20:24:31.5960097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:33.7653600Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:33.7654597Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:33.7655632Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:33.7656395Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:33.7721800Z Running test: large_conv_model sync interval 2022-03-17T20:24:33.7722332Z 3 with iteration mapping 2022-03-17T20:24:33.7722781Z {0: 0, 1: 10} 2022-03-17T20:24:34.8573994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:44.6350284Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:44.6351105Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:44.6352158Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:44.6352922Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:44.6417823Z Running test: large_conv_model sync interval 2022-03-17T20:24:44.6418200Z 3 with iteration mapping 2022-03-17T20:24:44.6418431Z {0: 5, 1: 7} 2022-03-17T20:24:45.7335680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:45.7336177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:52.2735407Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:52.2736204Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:52.2737214Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:24:52.2737984Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:24:52.2802548Z Running test: large_conv_model sync interval 2022-03-17T20:24:52.2802832Z 3 with iteration mapping 2022-03-17T20:24:52.2803104Z {0: 5, 1: 8} 2022-03-17T20:24:53.3803371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:24:53.3803892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:01.0214033Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:01.0215141Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:01.0216172Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:01.0217014Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:01.0282371Z Running test: large_conv_model sync interval 2022-03-17T20:25:01.0282766Z 3 with iteration mapping 2022-03-17T20:25:01.0282951Z {0: 5, 1: 15} 2022-03-17T20:25:02.1227900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:02.1228422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.3367394Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.3368247Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.3369285Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.3370062Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.3435457Z Running test: small_model sync interval 2022-03-17T20:25:17.3435773Z 4 with iteration mapping 2022-03-17T20:25:17.3436053Z {0: 0, 1: 2} 2022-03-17T20:25:17.3474152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.3548776Z Running test: small_model sync interval 2022-03-17T20:25:17.3549111Z 4 with iteration mapping 2022-03-17T20:25:17.3549364Z {0: 0, 1: 3} 2022-03-17T20:25:17.3586154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.3669448Z Running test: small_model sync interval 2022-03-17T20:25:17.3669755Z 4 with iteration mapping 2022-03-17T20:25:17.3670031Z {0: 0, 1: 10} 2022-03-17T20:25:17.3706061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.3866406Z Running test: small_model sync interval 2022-03-17T20:25:17.3866750Z 4 with iteration mapping 2022-03-17T20:25:17.3866989Z {0: 5, 1: 7} 2022-03-17T20:25:17.3905135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.3905627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.4032675Z Running test: small_model sync interval 2022-03-17T20:25:17.4032998Z 4 with iteration mapping 2022-03-17T20:25:17.4033247Z {0: 5, 1: 8} 2022-03-17T20:25:17.4070663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.4071157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.4206043Z Running test: small_model sync interval 2022-03-17T20:25:17.4206649Z 4 with iteration mapping 2022-03-17T20:25:17.4206922Z {0: 5, 1: 15} 2022-03-17T20:25:17.4243885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.4244377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.4458642Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.4459035Z 5 with iteration mapping 2022-03-17T20:25:17.4459288Z {0: 0, 1: 2} 2022-03-17T20:25:17.4502838Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4503674Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4504724Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4505482Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4524944Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.4525550Z 5 with iteration mapping 2022-03-17T20:25:17.4525809Z {0: 0, 1: 3} 2022-03-17T20:25:17.4576656Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4577436Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4578443Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4579200Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4599823Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.4600204Z 5 with iteration mapping 2022-03-17T20:25:17.4600449Z {0: 0, 1: 10} 2022-03-17T20:25:17.4712741Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4713494Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4714506Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4715284Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4734322Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.4735084Z 5 with iteration mapping 2022-03-17T20:25:17.4735300Z {0: 5, 1: 7} 2022-03-17T20:25:17.4823127Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4823842Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4824999Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4825776Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4845649Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.4846042Z 5 with iteration mapping 2022-03-17T20:25:17.4846318Z {0: 5, 1: 8} 2022-03-17T20:25:17.4942768Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4943509Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4944531Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.4945290Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.4965537Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.4966062Z 5 with iteration mapping 2022-03-17T20:25:17.4966357Z {0: 5, 1: 15} 2022-03-17T20:25:17.5122756Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5123548Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5124573Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5125331Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5145679Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.5146054Z 6 with iteration mapping 2022-03-17T20:25:17.5146279Z {0: 0, 1: 2} 2022-03-17T20:25:17.5190368Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5191297Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5192305Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5193069Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5212385Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.5213046Z 6 with iteration mapping 2022-03-17T20:25:17.5213337Z {0: 0, 1: 3} 2022-03-17T20:25:17.5266547Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5267452Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5268481Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5269241Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5288836Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.5289585Z 6 with iteration mapping 2022-03-17T20:25:17.5290093Z {0: 0, 1: 10} 2022-03-17T20:25:17.5409999Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5410793Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5411835Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5412583Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5432412Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.5433188Z 6 with iteration mapping 2022-03-17T20:25:17.5433674Z {0: 5, 1: 7} 2022-03-17T20:25:17.5525872Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5526674Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5527688Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5528440Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5548755Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.5549514Z 6 with iteration mapping 2022-03-17T20:25:17.5550019Z {0: 5, 1: 8} 2022-03-17T20:25:17.5651536Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5652412Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5653451Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5654214Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5674185Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:17.5674936Z 6 with iteration mapping 2022-03-17T20:25:17.5675461Z {0: 5, 1: 15} 2022-03-17T20:25:17.5842617Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5843409Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5844436Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:17.5845186Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:17.5864552Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:17.5865305Z 7 with iteration mapping 2022-03-17T20:25:17.5865765Z {0: 0, 1: 2} 2022-03-17T20:25:17.5901274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.5976233Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:17.5976996Z 7 with iteration mapping 2022-03-17T20:25:17.5977499Z {0: 0, 1: 3} 2022-03-17T20:25:17.6013071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6096540Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:17.6097277Z 7 with iteration mapping 2022-03-17T20:25:17.6097774Z {0: 0, 1: 10} 2022-03-17T20:25:17.6133434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6284496Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:17.6285194Z 7 with iteration mapping 2022-03-17T20:25:17.6285763Z {0: 5, 1: 7} 2022-03-17T20:25:17.6323132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6324141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6438798Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:17.6439737Z 7 with iteration mapping 2022-03-17T20:25:17.6440164Z {0: 5, 1: 8} 2022-03-17T20:25:17.6476637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6477687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6610546Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:17.6611335Z 7 with iteration mapping 2022-03-17T20:25:17.6611848Z {0: 5, 1: 15} 2022-03-17T20:25:17.6648370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6649498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6850276Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:17.6850965Z 8 with iteration mapping 2022-03-17T20:25:17.6851507Z {0: 0, 1: 2} 2022-03-17T20:25:17.6887271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.6962491Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:17.6963251Z 8 with iteration mapping 2022-03-17T20:25:17.6963744Z {0: 0, 1: 3} 2022-03-17T20:25:17.6999305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7083007Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:17.7083718Z 8 with iteration mapping 2022-03-17T20:25:17.7084245Z {0: 0, 1: 10} 2022-03-17T20:25:17.7119899Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7269843Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:17.7270609Z 8 with iteration mapping 2022-03-17T20:25:17.7271058Z {0: 5, 1: 7} 2022-03-17T20:25:17.7308159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7309212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7423779Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:17.7424543Z 8 with iteration mapping 2022-03-17T20:25:17.7424967Z {0: 5, 1: 8} 2022-03-17T20:25:17.7461427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7462444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7585487Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:17.7586219Z 8 with iteration mapping 2022-03-17T20:25:17.7586747Z {0: 5, 1: 15} 2022-03-17T20:25:17.7623493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7624516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7815095Z Running test: batch_norm_net sync interval 2022-03-17T20:25:17.7815765Z 1 with iteration mapping 2022-03-17T20:25:17.7816320Z {0: 0, 1: 2} 2022-03-17T20:25:17.7837872Z Running test: batch_norm_net sync interval 2022-03-17T20:25:17.7838536Z 1 with iteration mapping 2022-03-17T20:25:17.7839049Z {0: 0, 1: 3} 2022-03-17T20:25:17.7858003Z Running test: batch_norm_net sync interval 2022-03-17T20:25:17.7858721Z 1 with iteration mapping 2022-03-17T20:25:17.7859218Z {0: 0, 1: 10} 2022-03-17T20:25:17.7878169Z Running test: batch_norm_net sync interval 2022-03-17T20:25:17.7878892Z 1 with iteration mapping 2022-03-17T20:25:17.7879351Z {0: 5, 1: 7} 2022-03-17T20:25:17.7923096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.7924104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.8011071Z Running test: batch_norm_net sync interval 2022-03-17T20:25:17.8011760Z 1 with iteration mapping 2022-03-17T20:25:17.8012255Z {0: 5, 1: 8} 2022-03-17T20:25:17.8073587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.8075999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.8167985Z Running test: batch_norm_net sync interval 2022-03-17T20:25:17.8168333Z 1 with iteration mapping 2022-03-17T20:25:17.8168590Z {0: 5, 1: 15} 2022-03-17T20:25:17.8210883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.8211357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:17.8297864Z Running test: large_conv_model sync interval 2022-03-17T20:25:17.8298203Z 1 with iteration mapping 2022-03-17T20:25:17.8298439Z {0: 0, 1: 2} 2022-03-17T20:25:17.8318620Z Running test: large_conv_model sync interval 2022-03-17T20:25:17.8318978Z 1 with iteration mapping 2022-03-17T20:25:17.8319179Z {0: 0, 1: 3} 2022-03-17T20:25:17.8338747Z Running test: large_conv_model sync interval 2022-03-17T20:25:17.8339111Z 1 with iteration mapping 2022-03-17T20:25:17.8339382Z {0: 0, 1: 10} 2022-03-17T20:25:17.8359513Z Running test: large_conv_model sync interval 2022-03-17T20:25:17.8359876Z 1 with iteration mapping 2022-03-17T20:25:17.8360084Z {0: 5, 1: 7} 2022-03-17T20:25:18.9294715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:18.9295276Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:23.3078620Z Running test: large_conv_model sync interval 2022-03-17T20:25:23.3078981Z 1 with iteration mapping 2022-03-17T20:25:23.3079250Z {0: 5, 1: 8} 2022-03-17T20:25:24.4108327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:24.4108822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:28.7843227Z Running test: large_conv_model sync interval 2022-03-17T20:25:28.7843568Z 1 with iteration mapping 2022-03-17T20:25:28.7843855Z {0: 5, 1: 15} 2022-03-17T20:25:29.8864008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:29.8864536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.2482119Z Running test: small_model sync interval 2022-03-17T20:25:34.2482487Z 1 with iteration mapping 2022-03-17T20:25:34.2482729Z {0: 0, 1: 2} 2022-03-17T20:25:34.2500754Z Running test: small_model sync interval 2022-03-17T20:25:34.2501133Z 1 with iteration mapping 2022-03-17T20:25:34.2501349Z {0: 0, 1: 3} 2022-03-17T20:25:34.2519196Z Running test: small_model sync interval 2022-03-17T20:25:34.2519485Z 1 with iteration mapping 2022-03-17T20:25:34.2519739Z {0: 0, 1: 10} 2022-03-17T20:25:34.2537622Z Running test: small_model sync interval 2022-03-17T20:25:34.2537908Z 1 with iteration mapping 2022-03-17T20:25:34.2538158Z {0: 5, 1: 7} 2022-03-17T20:25:34.2580097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.2581206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.2668206Z Running test: small_model sync interval 2022-03-17T20:25:34.2668539Z 1 with iteration mapping 2022-03-17T20:25:34.2668773Z {0: 5, 1: 8} 2022-03-17T20:25:34.2709482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.2709969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.2795194Z Running test: small_model sync interval 2022-03-17T20:25:34.2795641Z 1 with iteration mapping 2022-03-17T20:25:34.2795889Z {0: 5, 1: 15} 2022-03-17T20:25:34.2836323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.2836812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.2923467Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.2923802Z 1 with iteration mapping 2022-03-17T20:25:34.2924071Z {0: 0, 1: 2} 2022-03-17T20:25:34.2941232Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.2941579Z 1 with iteration mapping 2022-03-17T20:25:34.2941830Z {0: 0, 1: 3} 2022-03-17T20:25:34.2959537Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.2959868Z 1 with iteration mapping 2022-03-17T20:25:34.2960124Z {0: 0, 1: 10} 2022-03-17T20:25:34.2977226Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.2977549Z 1 with iteration mapping 2022-03-17T20:25:34.2977815Z {0: 5, 1: 7} 2022-03-17T20:25:34.3064242Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.3064595Z 1 with iteration mapping 2022-03-17T20:25:34.3064832Z {0: 5, 1: 8} 2022-03-17T20:25:34.3150321Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.3150661Z 1 with iteration mapping 2022-03-17T20:25:34.3150941Z {0: 5, 1: 15} 2022-03-17T20:25:34.3236813Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.3237160Z 1 with iteration mapping 2022-03-17T20:25:34.3237407Z {0: 0, 1: 2} 2022-03-17T20:25:34.3254700Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.3255033Z 1 with iteration mapping 2022-03-17T20:25:34.3255300Z {0: 0, 1: 3} 2022-03-17T20:25:34.3273415Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.3273748Z 1 with iteration mapping 2022-03-17T20:25:34.3274017Z {0: 0, 1: 10} 2022-03-17T20:25:34.3291045Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.3291375Z 1 with iteration mapping 2022-03-17T20:25:34.3291644Z {0: 5, 1: 7} 2022-03-17T20:25:34.3376590Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.3376982Z 1 with iteration mapping 2022-03-17T20:25:34.3377252Z {0: 5, 1: 8} 2022-03-17T20:25:34.3470406Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:25:34.3470814Z 1 with iteration mapping 2022-03-17T20:25:34.3471083Z {0: 5, 1: 15} 2022-03-17T20:25:34.3556682Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:34.3557070Z 1 with iteration mapping 2022-03-17T20:25:34.3557448Z {0: 0, 1: 2} 2022-03-17T20:25:34.3575520Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:34.3575883Z 1 with iteration mapping 2022-03-17T20:25:34.3576130Z {0: 0, 1: 3} 2022-03-17T20:25:34.3594300Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:34.3594665Z 1 with iteration mapping 2022-03-17T20:25:34.3594925Z {0: 0, 1: 10} 2022-03-17T20:25:34.3612955Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:34.3613325Z 1 with iteration mapping 2022-03-17T20:25:34.3613584Z {0: 5, 1: 7} 2022-03-17T20:25:34.3653936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.3654447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.3739822Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:34.3740189Z 1 with iteration mapping 2022-03-17T20:25:34.3740450Z {0: 5, 1: 8} 2022-03-17T20:25:34.3781428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.3781912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.3868472Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:25:34.3868838Z 1 with iteration mapping 2022-03-17T20:25:34.3869096Z {0: 5, 1: 15} 2022-03-17T20:25:34.3909697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.3910160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.3995915Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:34.3996231Z 1 with iteration mapping 2022-03-17T20:25:34.3996486Z {0: 0, 1: 2} 2022-03-17T20:25:34.4014368Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:34.4014726Z 1 with iteration mapping 2022-03-17T20:25:34.4014996Z {0: 0, 1: 3} 2022-03-17T20:25:34.4032988Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:34.4033365Z 1 with iteration mapping 2022-03-17T20:25:34.4033635Z {0: 0, 1: 10} 2022-03-17T20:25:34.4051612Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:34.4052009Z 1 with iteration mapping 2022-03-17T20:25:34.4052259Z {0: 5, 1: 7} 2022-03-17T20:25:34.4093179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.4093609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.4179225Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:34.4179561Z 1 with iteration mapping 2022-03-17T20:25:34.4179829Z {0: 5, 1: 8} 2022-03-17T20:25:34.4220479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.4220981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.4307860Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:25:34.4308204Z 1 with iteration mapping 2022-03-17T20:25:34.4308464Z {0: 5, 1: 15} 2022-03-17T20:25:34.4349378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.4349876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:34.9649465Z ok (122.782s) 2022-03-17T20:25:34.9649613Z 2022-03-17T20:25:34.9650052Z ---------------------------------------------------------------------- 2022-03-17T20:25:34.9650619Z Ran 1 test in 122.783s 2022-03-17T20:25:34.9650786Z 2022-03-17T20:25:34.9650868Z OK 2022-03-17T20:25:34.9650996Z 2022-03-17T20:25:34.9651138Z Generating XML reports... 2022-03-17T20:25:34.9696665Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202332.xml 2022-03-17T20:25:36.2642355Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:36.2657355Z 2022-03-17T20:25:36.2657606Z Running tests... 2022-03-17T20:25:36.2658108Z ---------------------------------------------------------------------- 2022-03-17T20:25:37.8252519Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:25:37.8674307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123688 2022-03-17T20:25:37.8802589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123689 2022-03-17T20:25:38.9427922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:25:38.9784843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:25:38.9785630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:38.9832868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:38.9839600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:25:39.0799990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:25:40.6980500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:40.7093995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:40.7095200Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:40.7096107Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:40.7097124Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:40.7097899Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:40.7194031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:40.7194514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:25:40.7241267Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:25:40.7242060Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:25:40.9880110Z ok (4.722s) 2022-03-17T20:25:40.9880358Z 2022-03-17T20:25:40.9880741Z ---------------------------------------------------------------------- 2022-03-17T20:25:40.9881074Z Ran 1 test in 4.722s 2022-03-17T20:25:40.9881222Z 2022-03-17T20:25:40.9881321Z OK 2022-03-17T20:25:40.9881456Z 2022-03-17T20:25:40.9881843Z Generating XML reports... 2022-03-17T20:25:40.9925231Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202536.xml 2022-03-17T20:25:42.2937207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:42.2952896Z 2022-03-17T20:25:42.2953121Z Running tests... 2022-03-17T20:25:42.2953564Z ---------------------------------------------------------------------- 2022-03-17T20:25:43.8387769Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:25:43.8810497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123814 2022-03-17T20:25:43.8935186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123815 2022-03-17T20:25:44.9318957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:25:44.9820206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:25:44.9821246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:44.9825376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:44.9832307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:25:45.0836036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:25:48.1049466Z ok (5.809s) 2022-03-17T20:25:48.1049745Z 2022-03-17T20:25:48.1050164Z ---------------------------------------------------------------------- 2022-03-17T20:25:48.1050438Z Ran 1 test in 5.810s 2022-03-17T20:25:48.1050620Z 2022-03-17T20:25:48.1050676Z OK 2022-03-17T20:25:48.1050871Z 2022-03-17T20:25:48.1050972Z Generating XML reports... 2022-03-17T20:25:48.1096932Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202542.xml 2022-03-17T20:25:49.4058014Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:49.4073339Z 2022-03-17T20:25:49.4073543Z Running tests... 2022-03-17T20:25:49.4074038Z ---------------------------------------------------------------------- 2022-03-17T20:25:50.9383669Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:25:50.9808982Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 123941 2022-03-17T20:25:50.9936652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 123942 2022-03-17T20:25:52.0370758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:25:52.0722124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:25:52.0722900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:52.0776671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:52.0783857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:25:52.0786943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:25:52.1734099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:25:52.1738579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:25:52.1739277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:25:52.1806262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:25:52.3983348Z ok (2.991s) 2022-03-17T20:25:52.3983836Z 2022-03-17T20:25:52.3984388Z ---------------------------------------------------------------------- 2022-03-17T20:25:52.3984773Z Ran 1 test in 2.991s 2022-03-17T20:25:52.3984937Z 2022-03-17T20:25:52.3985040Z OK 2022-03-17T20:25:52.3985173Z 2022-03-17T20:25:52.3985271Z Generating XML reports... 2022-03-17T20:25:52.4029580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202549.xml 2022-03-17T20:25:53.6791974Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:53.6807491Z 2022-03-17T20:25:53.6807933Z Running tests... 2022-03-17T20:25:53.6808434Z ---------------------------------------------------------------------- 2022-03-17T20:25:55.2148143Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:25:55.2572069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124058 2022-03-17T20:25:55.2698967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124059 2022-03-17T20:25:56.3115982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:25:56.3323908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:25:56.3324673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:56.3420140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:25:56.3427384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:25:56.3430954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:25:56.4336260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:25:56.4339947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:25:56.4340816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:25:56.4347288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:25:56.5745249Z ok (2.893s) 2022-03-17T20:25:56.5745520Z 2022-03-17T20:25:56.5746120Z ---------------------------------------------------------------------- 2022-03-17T20:25:56.5746392Z Ran 1 test in 2.894s 2022-03-17T20:25:56.5746506Z 2022-03-17T20:25:56.5746595Z OK 2022-03-17T20:25:56.5746728Z 2022-03-17T20:25:56.5746840Z Generating XML reports... 2022-03-17T20:25:56.5790972Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202553.xml 2022-03-17T20:25:57.8576068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:25:57.8591669Z 2022-03-17T20:25:57.8591870Z Running tests... 2022-03-17T20:25:57.8592327Z ---------------------------------------------------------------------- 2022-03-17T20:25:59.3869912Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:25:59.4295038Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124175 2022-03-17T20:25:59.4422692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124176 2022-03-17T20:26:00.5127107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:00.5315314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:00.5316334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:00.5329591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:00.5336506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:00.6331421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:02.2715604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:02.2716375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:02.2773261Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:26:02.2774825Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:26:02.5499839Z ok (4.690s) 2022-03-17T20:26:02.5504588Z 2022-03-17T20:26:02.5505595Z ---------------------------------------------------------------------- 2022-03-17T20:26:02.5506257Z Ran 1 test in 4.691s 2022-03-17T20:26:02.5506559Z 2022-03-17T20:26:02.5506719Z OK 2022-03-17T20:26:02.5506910Z 2022-03-17T20:26:02.5507109Z Generating XML reports... 2022-03-17T20:26:02.5550491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202557.xml 2022-03-17T20:26:03.8441871Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:03.8457046Z 2022-03-17T20:26:03.8457277Z Running tests... 2022-03-17T20:26:03.8457730Z ---------------------------------------------------------------------- 2022-03-17T20:26:05.3851632Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:05.4272298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124301 2022-03-17T20:26:05.4403412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124302 2022-03-17T20:26:06.5552436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:06.5614656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:06.5615818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:06.5653744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:06.5660167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:06.6630866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:08.2743177Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:26:08.2896297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:08.2897222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:26:08.5483666Z ok (4.702s) 2022-03-17T20:26:08.5483947Z 2022-03-17T20:26:08.5484516Z ---------------------------------------------------------------------- 2022-03-17T20:26:08.5484804Z Ran 1 test in 4.703s 2022-03-17T20:26:08.5484971Z 2022-03-17T20:26:08.5485062Z OK 2022-03-17T20:26:08.5485196Z 2022-03-17T20:26:08.5485406Z Generating XML reports... 2022-03-17T20:26:08.5528433Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202603.xml 2022-03-17T20:26:09.8558179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:09.8573692Z 2022-03-17T20:26:09.8574103Z Running tests... 2022-03-17T20:26:09.8574545Z ---------------------------------------------------------------------- 2022-03-17T20:26:11.3987507Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:11.4412906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124427 2022-03-17T20:26:11.4537925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124428 2022-03-17T20:26:12.5526217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:12.5597022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:12.5597845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:12.5627556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:12.5634050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:12.6609360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:12.8588167Z ok (3.001s) 2022-03-17T20:26:12.8588360Z 2022-03-17T20:26:12.8588790Z ---------------------------------------------------------------------- 2022-03-17T20:26:12.8589099Z Ran 1 test in 3.001s 2022-03-17T20:26:12.8589266Z 2022-03-17T20:26:12.8589364Z OK 2022-03-17T20:26:12.8589504Z 2022-03-17T20:26:12.8589613Z Generating XML reports... 2022-03-17T20:26:12.8636691Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202609.xml 2022-03-17T20:26:14.1520238Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:14.1535533Z 2022-03-17T20:26:14.1535753Z Running tests... 2022-03-17T20:26:14.1536167Z ---------------------------------------------------------------------- 2022-03-17T20:26:14.1555677Z test_gather (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:26:14.1555877Z 2022-03-17T20:26:14.1556217Z ---------------------------------------------------------------------- 2022-03-17T20:26:14.1556550Z Ran 1 test in 0.002s 2022-03-17T20:26:14.1556682Z 2022-03-17T20:26:14.1556757Z OK (skipped=1) 2022-03-17T20:26:14.1556896Z 2022-03-17T20:26:14.1557041Z Generating XML reports... 2022-03-17T20:26:14.1601454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202614.xml 2022-03-17T20:26:15.2881953Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:15.2897211Z 2022-03-17T20:26:15.2897604Z Running tests... 2022-03-17T20:26:15.2898112Z ---------------------------------------------------------------------- 2022-03-17T20:26:15.2931182Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:26:15.2931421Z 2022-03-17T20:26:15.2931764Z ---------------------------------------------------------------------- 2022-03-17T20:26:15.2932101Z Ran 1 test in 0.003s 2022-03-17T20:26:15.2932271Z 2022-03-17T20:26:15.2932366Z OK (skipped=1) 2022-03-17T20:26:15.2932508Z 2022-03-17T20:26:15.2932577Z Generating XML reports... 2022-03-17T20:26:15.2982070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202615.xml 2022-03-17T20:26:16.3884861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:16.3901414Z 2022-03-17T20:26:16.3901733Z Running tests... 2022-03-17T20:26:16.3902240Z ---------------------------------------------------------------------- 2022-03-17T20:26:17.9440102Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:17.9866253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124610 2022-03-17T20:26:17.9995095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124611 2022-03-17T20:26:19.0926915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:19.0985279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:19.0986105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:19.1030303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:19.1037271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:19.2000540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:23.1104252Z ok (6.720s) 2022-03-17T20:26:23.1104992Z 2022-03-17T20:26:23.1105583Z ---------------------------------------------------------------------- 2022-03-17T20:26:23.1106169Z Ran 1 test in 6.720s 2022-03-17T20:26:23.1106384Z 2022-03-17T20:26:23.1106538Z OK 2022-03-17T20:26:23.1106776Z 2022-03-17T20:26:23.1106977Z Generating XML reports... 2022-03-17T20:26:23.1153541Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202616.xml 2022-03-17T20:26:24.4069925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:24.4084985Z 2022-03-17T20:26:24.4085255Z Running tests... 2022-03-17T20:26:24.4085685Z ---------------------------------------------------------------------- 2022-03-17T20:26:24.4105177Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:26:24.4105381Z 2022-03-17T20:26:24.4105705Z ---------------------------------------------------------------------- 2022-03-17T20:26:24.4106295Z Ran 1 test in 0.002s 2022-03-17T20:26:24.4106465Z 2022-03-17T20:26:24.4106546Z OK (skipped=1) 2022-03-17T20:26:24.4106735Z 2022-03-17T20:26:24.4106771Z Generating XML reports... 2022-03-17T20:26:24.4150446Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202624.xml 2022-03-17T20:26:25.5110358Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:25.5125637Z 2022-03-17T20:26:25.5125834Z Running tests... 2022-03-17T20:26:25.5126300Z ---------------------------------------------------------------------- 2022-03-17T20:26:25.5146177Z test_gather_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:26:25.5146452Z 2022-03-17T20:26:25.5146993Z ---------------------------------------------------------------------- 2022-03-17T20:26:25.5147368Z Ran 1 test in 0.002s 2022-03-17T20:26:25.5147520Z 2022-03-17T20:26:25.5147622Z OK (skipped=1) 2022-03-17T20:26:25.5147766Z 2022-03-17T20:26:25.5147856Z Generating XML reports... 2022-03-17T20:26:25.5191446Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202625.xml 2022-03-17T20:26:26.6570367Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:26.6585446Z 2022-03-17T20:26:26.6585685Z Running tests... 2022-03-17T20:26:26.6586394Z ---------------------------------------------------------------------- 2022-03-17T20:26:28.2090592Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:28.2510238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124808 2022-03-17T20:26:28.2635721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124809 2022-03-17T20:26:29.3431274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:29.3774262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:29.3775278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:29.3837663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:29.3844044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:29.4790441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:32.2732064Z ok (5.614s) 2022-03-17T20:26:32.2732301Z 2022-03-17T20:26:32.2733010Z ---------------------------------------------------------------------- 2022-03-17T20:26:32.2733436Z Ran 1 test in 5.615s 2022-03-17T20:26:32.2733613Z 2022-03-17T20:26:32.2733696Z OK 2022-03-17T20:26:32.2733830Z 2022-03-17T20:26:32.2733944Z Generating XML reports... 2022-03-17T20:26:32.2778879Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202626.xml 2022-03-17T20:26:33.5809173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:33.5824827Z 2022-03-17T20:26:33.5824928Z Running tests... 2022-03-17T20:26:33.5825914Z ---------------------------------------------------------------------- 2022-03-17T20:26:35.1145155Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:35.1569322Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 124932 2022-03-17T20:26:35.1694271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 124933 2022-03-17T20:26:36.2319266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:36.2620501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:36.2621817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:36.2623363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:36.2630261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:36.2631444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:36.2843873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:26:36.2844658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:26:36.2845581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:26:36.2846247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:26:38.7590087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:26:38.7590608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:26:38.7591634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:26:38.7592324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:26:38.8044782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:26:38.8045346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:26:38.8046103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:26:38.8046793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:26:39.2790846Z ok (5.696s) 2022-03-17T20:26:39.2791014Z 2022-03-17T20:26:39.2791413Z ---------------------------------------------------------------------- 2022-03-17T20:26:39.2791709Z Ran 1 test in 5.697s 2022-03-17T20:26:39.2791873Z 2022-03-17T20:26:39.2791960Z OK 2022-03-17T20:26:39.2792074Z 2022-03-17T20:26:39.2792196Z Generating XML reports... 2022-03-17T20:26:39.2837336Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202633.xml 2022-03-17T20:26:40.5868527Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:40.5883600Z 2022-03-17T20:26:40.5883890Z Running tests... 2022-03-17T20:26:40.5884655Z ---------------------------------------------------------------------- 2022-03-17T20:26:42.1168850Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:42.1591696Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125095 2022-03-17T20:26:42.1717900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125096 2022-03-17T20:26:43.2042682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:43.2433431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:43.2434152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:43.2447935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:43.2454767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:43.2458112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:26:43.3444933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:43.3448324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:26:43.3448994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:26:43.3478033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:26:43.5766705Z ok (2.988s) 2022-03-17T20:26:43.5766961Z 2022-03-17T20:26:43.5767647Z ---------------------------------------------------------------------- 2022-03-17T20:26:43.5767918Z Ran 1 test in 2.988s 2022-03-17T20:26:43.5768043Z 2022-03-17T20:26:43.5768134Z OK 2022-03-17T20:26:43.5768270Z 2022-03-17T20:26:43.5768408Z Generating XML reports... 2022-03-17T20:26:43.5813374Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202640.xml 2022-03-17T20:26:44.8573106Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:44.8588665Z 2022-03-17T20:26:44.8589005Z Running tests... 2022-03-17T20:26:44.8589403Z ---------------------------------------------------------------------- 2022-03-17T20:26:46.3969699Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:46.4397745Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125212 2022-03-17T20:26:46.4524351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125213 2022-03-17T20:26:47.4890333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:47.5363679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:47.5364929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:47.5397444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:47.5403986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:47.6378605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:50.1614155Z ok (5.302s) 2022-03-17T20:26:50.1614369Z 2022-03-17T20:26:50.1614777Z ---------------------------------------------------------------------- 2022-03-17T20:26:50.1615151Z Ran 1 test in 5.302s 2022-03-17T20:26:50.1615301Z 2022-03-17T20:26:50.1615335Z OK 2022-03-17T20:26:50.1615453Z 2022-03-17T20:26:50.1615587Z Generating XML reports... 2022-03-17T20:26:50.1662014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202644.xml 2022-03-17T20:26:51.4615346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:51.4630755Z 2022-03-17T20:26:51.4631068Z Running tests... 2022-03-17T20:26:51.4631501Z ---------------------------------------------------------------------- 2022-03-17T20:26:52.9995548Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:53.0416619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125335 2022-03-17T20:26:53.0540360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125336 2022-03-17T20:26:54.1291527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:54.1542285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:54.1543413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:54.1595674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:54.1602564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:54.2555199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:54.5592038Z ok (3.096s) 2022-03-17T20:26:54.5592260Z 2022-03-17T20:26:54.5592710Z ---------------------------------------------------------------------- 2022-03-17T20:26:54.5592978Z Ran 1 test in 3.096s 2022-03-17T20:26:54.5593145Z 2022-03-17T20:26:54.5593519Z OK 2022-03-17T20:26:54.5593658Z 2022-03-17T20:26:54.5593789Z Generating XML reports... 2022-03-17T20:26:54.5638372Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202651.xml 2022-03-17T20:26:55.8407379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:26:55.8422639Z 2022-03-17T20:26:55.8423109Z Running tests... 2022-03-17T20:26:55.8423600Z ---------------------------------------------------------------------- 2022-03-17T20:26:57.3899297Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:26:57.4327999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125448 2022-03-17T20:26:57.4453389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125449 2022-03-17T20:26:58.4825706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:26:58.5284540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:26:58.5285518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:58.5331530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:26:58.5338277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:26:58.5341411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:26:58.6298230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:26:58.6301821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:26:58.6302522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:26:58.6359923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:26:58.8503495Z ok (3.008s) 2022-03-17T20:26:58.8503730Z 2022-03-17T20:26:58.8504109Z ---------------------------------------------------------------------- 2022-03-17T20:26:58.8504449Z Ran 1 test in 3.008s 2022-03-17T20:26:58.8504614Z 2022-03-17T20:26:58.8504694Z OK 2022-03-17T20:26:58.8504831Z 2022-03-17T20:26:58.8504963Z Generating XML reports... 2022-03-17T20:26:58.8549011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202655.xml 2022-03-17T20:27:00.1325477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:00.1340610Z 2022-03-17T20:27:00.1340926Z Running tests... 2022-03-17T20:27:00.1341346Z ---------------------------------------------------------------------- 2022-03-17T20:27:01.6865337Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:27:01.7309379Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125565 2022-03-17T20:27:01.7450144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125566 2022-03-17T20:27:02.7879635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:27:02.8204981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:27:02.8205785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:02.8284931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:02.8292540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:27:02.8296003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:27:02.9217823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:27:02.9222633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:27:02.9223288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:27:02.9316970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:27:03.1499825Z ok (3.016s) 2022-03-17T20:27:03.1500588Z 2022-03-17T20:27:03.1501176Z ---------------------------------------------------------------------- 2022-03-17T20:27:03.1501811Z Ran 1 test in 3.016s 2022-03-17T20:27:03.1502069Z 2022-03-17T20:27:03.1502239Z OK 2022-03-17T20:27:03.1502521Z 2022-03-17T20:27:03.1502733Z Generating XML reports... 2022-03-17T20:27:03.1546901Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202700.xml 2022-03-17T20:27:04.4364276Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:04.4379340Z 2022-03-17T20:27:04.4379557Z Running tests... 2022-03-17T20:27:04.4380020Z ---------------------------------------------------------------------- 2022-03-17T20:27:05.9827346Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:27:06.0251422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125682 2022-03-17T20:27:06.0378285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125683 2022-03-17T20:27:07.1246919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:27:07.1277766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:27:07.1278914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:07.1348283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:07.1355199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:27:07.2292599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:27:09.1456548Z ok (4.707s) 2022-03-17T20:27:09.1457010Z 2022-03-17T20:27:09.1457365Z ---------------------------------------------------------------------- 2022-03-17T20:27:09.1457683Z Ran 1 test in 4.708s 2022-03-17T20:27:09.1457845Z 2022-03-17T20:27:09.1457936Z OK 2022-03-17T20:27:09.1458070Z 2022-03-17T20:27:09.1458289Z Generating XML reports... 2022-03-17T20:27:09.1503623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202704.xml 2022-03-17T20:27:10.4478753Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:10.4493856Z 2022-03-17T20:27:10.4494305Z Running tests... 2022-03-17T20:27:10.4494790Z ---------------------------------------------------------------------- 2022-03-17T20:27:10.4522019Z test_irecv (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:27:10.4522193Z 2022-03-17T20:27:10.4522623Z ---------------------------------------------------------------------- 2022-03-17T20:27:10.4523177Z Ran 1 test in 0.003s 2022-03-17T20:27:10.4524156Z 2022-03-17T20:27:10.4524618Z OK (skipped=1) 2022-03-17T20:27:10.4524807Z 2022-03-17T20:27:10.4524935Z Generating XML reports... 2022-03-17T20:27:10.4567199Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202710.xml 2022-03-17T20:27:11.5628501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:11.5643720Z 2022-03-17T20:27:11.5644098Z Running tests... 2022-03-17T20:27:11.5644589Z ---------------------------------------------------------------------- 2022-03-17T20:27:11.5663760Z test_isend (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:27:11.5664046Z 2022-03-17T20:27:11.5664594Z ---------------------------------------------------------------------- 2022-03-17T20:27:11.5665284Z Ran 1 test in 0.002s 2022-03-17T20:27:11.5665556Z 2022-03-17T20:27:11.5665670Z OK (skipped=1) 2022-03-17T20:27:11.5665827Z 2022-03-17T20:27:11.5666255Z Generating XML reports... 2022-03-17T20:27:11.5709562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202711.xml 2022-03-17T20:27:12.7233660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:12.7248708Z 2022-03-17T20:27:12.7248994Z Running tests... 2022-03-17T20:27:12.7249421Z ---------------------------------------------------------------------- 2022-03-17T20:27:12.7269039Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:27:12.7269327Z 2022-03-17T20:27:12.7269616Z ---------------------------------------------------------------------- 2022-03-17T20:27:12.7269927Z Ran 1 test in 0.002s 2022-03-17T20:27:12.7270095Z 2022-03-17T20:27:12.7270201Z OK (skipped=1) 2022-03-17T20:27:12.7270359Z 2022-03-17T20:27:12.7270461Z Generating XML reports... 2022-03-17T20:27:12.7314165Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202712.xml 2022-03-17T20:27:13.8681659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:13.8696545Z 2022-03-17T20:27:13.8696834Z Running tests... 2022-03-17T20:27:13.8697256Z ---------------------------------------------------------------------- 2022-03-17T20:27:13.8717557Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:27:13.8717909Z 2022-03-17T20:27:13.8718198Z ---------------------------------------------------------------------- 2022-03-17T20:27:13.8718564Z Ran 1 test in 0.002s 2022-03-17T20:27:13.8718726Z 2022-03-17T20:27:13.8718838Z OK (skipped=1) 2022-03-17T20:27:13.8718982Z 2022-03-17T20:27:13.8719111Z Generating XML reports... 2022-03-17T20:27:13.8763124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202713.xml 2022-03-17T20:27:15.0136056Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:15.0151649Z 2022-03-17T20:27:15.0152037Z Running tests... 2022-03-17T20:27:15.0152599Z ---------------------------------------------------------------------- 2022-03-17T20:27:16.5671787Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:27:16.6095700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 125948 2022-03-17T20:27:16.6222341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 125949 2022-03-17T20:27:17.7003591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:27:17.7163127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:27:17.7163924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:17.7205766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:17.7212244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:27:17.7215500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:27:17.8176674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:27:17.8180981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:27:17.8181966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:27:17.8235521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:27:17.8393425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:27:17.8394190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:27:17.8394876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:27:17.8395550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:27:20.2775247Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T20:27:35.9569385Z ok (20.941s) 2022-03-17T20:27:35.9569623Z 2022-03-17T20:27:35.9570025Z ---------------------------------------------------------------------- 2022-03-17T20:27:35.9570322Z Ran 1 test in 20.942s 2022-03-17T20:27:35.9573296Z 2022-03-17T20:27:35.9573616Z OK 2022-03-17T20:27:35.9573797Z 2022-03-17T20:27:35.9573943Z Generating XML reports... 2022-03-17T20:27:35.9615535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202715.xml 2022-03-17T20:27:37.2465686Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:37.2480951Z 2022-03-17T20:27:37.2481087Z Running tests... 2022-03-17T20:27:37.2481833Z ---------------------------------------------------------------------- 2022-03-17T20:27:38.7672521Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:27:38.8093019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126079 2022-03-17T20:27:38.8219390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126080 2022-03-17T20:27:39.8628058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:27:39.8638765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:27:39.8639584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:39.8729790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:27:39.8736436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:27:39.8739640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:27:39.9649988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:27:39.9653999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:27:39.9654632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:27:39.9656052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:27:39.9864798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:27:39.9865291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:27:39.9866226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:27:39.9866910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:27:42.4251865Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T20:27:42.4252423Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T20:27:58.0602404Z ok (20.812s) 2022-03-17T20:27:58.0602661Z 2022-03-17T20:27:58.0603348Z ---------------------------------------------------------------------- 2022-03-17T20:27:58.0603735Z Ran 1 test in 20.812s 2022-03-17T20:27:58.0604699Z 2022-03-17T20:27:58.0605124Z OK 2022-03-17T20:27:58.0605325Z 2022-03-17T20:27:58.0605467Z Generating XML reports... 2022-03-17T20:27:58.0650281Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202737.xml 2022-03-17T20:27:59.3666537Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:27:59.3681524Z 2022-03-17T20:27:59.3681722Z Running tests... 2022-03-17T20:27:59.3682477Z ---------------------------------------------------------------------- 2022-03-17T20:27:59.3706849Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:27:59.3707143Z 2022-03-17T20:27:59.3707428Z ---------------------------------------------------------------------- 2022-03-17T20:27:59.3707765Z Ran 1 test in 0.003s 2022-03-17T20:27:59.3707913Z 2022-03-17T20:27:59.3708034Z OK (skipped=1) 2022-03-17T20:27:59.3708189Z 2022-03-17T20:27:59.3708314Z Generating XML reports... 2022-03-17T20:27:59.3751653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202759.xml 2022-03-17T20:28:00.5262404Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:00.5277682Z 2022-03-17T20:28:00.5277970Z Running tests... 2022-03-17T20:28:00.5278418Z ---------------------------------------------------------------------- 2022-03-17T20:28:00.5307673Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:28:00.5308145Z 2022-03-17T20:28:00.5308587Z ---------------------------------------------------------------------- 2022-03-17T20:28:00.5308865Z Ran 1 test in 0.003s 2022-03-17T20:28:00.5309010Z 2022-03-17T20:28:00.5309131Z OK (skipped=1) 2022-03-17T20:28:00.5309277Z 2022-03-17T20:28:00.5309419Z Generating XML reports... 2022-03-17T20:28:00.5353235Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202800.xml 2022-03-17T20:28:01.6801537Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:01.6817062Z 2022-03-17T20:28:01.6817515Z Running tests... 2022-03-17T20:28:01.6817979Z ---------------------------------------------------------------------- 2022-03-17T20:28:01.6840015Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:28:01.6840366Z 2022-03-17T20:28:01.6840914Z ---------------------------------------------------------------------- 2022-03-17T20:28:01.6841200Z Ran 1 test in 0.002s 2022-03-17T20:28:01.6841351Z 2022-03-17T20:28:01.6841408Z OK (skipped=1) 2022-03-17T20:28:01.6841564Z 2022-03-17T20:28:01.6841686Z Generating XML reports... 2022-03-17T20:28:01.6885447Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202801.xml 2022-03-17T20:28:02.7904376Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:02.7925663Z 2022-03-17T20:28:02.7926067Z Running tests... 2022-03-17T20:28:02.7926916Z ---------------------------------------------------------------------- 2022-03-17T20:28:02.7951355Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:28:02.7951989Z 2022-03-17T20:28:02.7952458Z ---------------------------------------------------------------------- 2022-03-17T20:28:02.7953115Z Ran 1 test in 0.003s 2022-03-17T20:28:02.7953418Z 2022-03-17T20:28:02.7953619Z OK (skipped=1) 2022-03-17T20:28:02.7955042Z 2022-03-17T20:28:02.7955392Z Generating XML reports... 2022-03-17T20:28:02.7999838Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202802.xml 2022-03-17T20:28:03.9455631Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:03.9470911Z 2022-03-17T20:28:03.9471408Z Running tests... 2022-03-17T20:28:03.9471872Z ---------------------------------------------------------------------- 2022-03-17T20:28:03.9493623Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:28:03.9493929Z 2022-03-17T20:28:03.9494218Z ---------------------------------------------------------------------- 2022-03-17T20:28:03.9494529Z Ran 1 test in 0.002s 2022-03-17T20:28:03.9494694Z 2022-03-17T20:28:03.9494722Z OK (skipped=1) 2022-03-17T20:28:03.9494878Z 2022-03-17T20:28:03.9495009Z Generating XML reports... 2022-03-17T20:28:03.9537802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202803.xml 2022-03-17T20:28:05.0937006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:05.0952531Z 2022-03-17T20:28:05.0952716Z Running tests... 2022-03-17T20:28:05.0953212Z ---------------------------------------------------------------------- 2022-03-17T20:28:06.6557717Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:06.6979410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126385 2022-03-17T20:28:06.7106818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126386 2022-03-17T20:28:07.7570523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:07.8087593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:07.8088626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:07.8177725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:07.8184794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:07.9103207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:09.5192229Z ok (4.424s) 2022-03-17T20:28:09.5192458Z 2022-03-17T20:28:09.5192848Z ---------------------------------------------------------------------- 2022-03-17T20:28:09.5193174Z Ran 1 test in 4.424s 2022-03-17T20:28:09.5193319Z 2022-03-17T20:28:09.5193411Z OK 2022-03-17T20:28:09.5193526Z 2022-03-17T20:28:09.5193669Z Generating XML reports... 2022-03-17T20:28:09.5238772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202805.xml 2022-03-17T20:28:10.8098917Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:10.8114955Z 2022-03-17T20:28:10.8115096Z Running tests... 2022-03-17T20:28:10.8115603Z ---------------------------------------------------------------------- 2022-03-17T20:28:12.3429227Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:12.3851761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126507 2022-03-17T20:28:12.3975545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126508 2022-03-17T20:28:13.4589657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:13.4646414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:13.4647194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:13.4691056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:13.4698034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:13.5660800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:15.2048728Z ok (4.393s) 2022-03-17T20:28:15.2049169Z 2022-03-17T20:28:15.2049762Z ---------------------------------------------------------------------- 2022-03-17T20:28:15.2050233Z Ran 1 test in 4.393s 2022-03-17T20:28:15.2050389Z 2022-03-17T20:28:15.2050469Z OK 2022-03-17T20:28:15.2050585Z 2022-03-17T20:28:15.2050719Z Generating XML reports... 2022-03-17T20:28:15.2093422Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202810.xml 2022-03-17T20:28:16.5045386Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:16.5060007Z 2022-03-17T20:28:16.5060239Z Running tests... 2022-03-17T20:28:16.5060977Z ---------------------------------------------------------------------- 2022-03-17T20:28:18.0381266Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:18.0805680Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126629 2022-03-17T20:28:18.0931985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126630 2022-03-17T20:28:19.1385311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:19.1398304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:19.1399102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:19.1486366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:19.1492974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:19.2413626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:21.9023363Z ok (5.396s) 2022-03-17T20:28:21.9023617Z 2022-03-17T20:28:21.9024014Z ---------------------------------------------------------------------- 2022-03-17T20:28:21.9024381Z Ran 1 test in 5.396s 2022-03-17T20:28:21.9024534Z 2022-03-17T20:28:21.9024632Z OK 2022-03-17T20:28:21.9024774Z 2022-03-17T20:28:21.9024916Z Generating XML reports... 2022-03-17T20:28:21.9071313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202816.xml 2022-03-17T20:28:23.2045429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:23.2069816Z 2022-03-17T20:28:23.2070116Z Running tests... 2022-03-17T20:28:23.2070547Z ---------------------------------------------------------------------- 2022-03-17T20:28:24.7632355Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:24.8058205Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126752 2022-03-17T20:28:24.8185969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126753 2022-03-17T20:28:25.8595096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:25.8934829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:25.8935616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:25.9002545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:25.9009879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:25.9951727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:27.6259329Z ok (4.419s) 2022-03-17T20:28:27.6259562Z 2022-03-17T20:28:27.6259963Z ---------------------------------------------------------------------- 2022-03-17T20:28:27.6260538Z Ran 1 test in 4.419s 2022-03-17T20:28:27.6260736Z 2022-03-17T20:28:27.6260830Z OK 2022-03-17T20:28:27.6260973Z 2022-03-17T20:28:27.6261147Z Generating XML reports... 2022-03-17T20:28:27.6305436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202823.xml 2022-03-17T20:28:28.9257864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:28.9274727Z 2022-03-17T20:28:28.9275253Z Running tests... 2022-03-17T20:28:28.9275745Z ---------------------------------------------------------------------- 2022-03-17T20:28:30.4571805Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:30.4992805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 126874 2022-03-17T20:28:30.5117514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 126875 2022-03-17T20:28:31.5865426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:31.5895858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:31.5896635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:31.5967642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:31.5974783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:31.6907897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:34.9216302Z ok (5.994s) 2022-03-17T20:28:34.9216549Z 2022-03-17T20:28:34.9216936Z ---------------------------------------------------------------------- 2022-03-17T20:28:34.9217277Z Ran 1 test in 5.994s 2022-03-17T20:28:34.9217445Z 2022-03-17T20:28:34.9217541Z OK 2022-03-17T20:28:34.9217679Z 2022-03-17T20:28:34.9217810Z Generating XML reports... 2022-03-17T20:28:34.9261371Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202828.xml 2022-03-17T20:28:36.2336865Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:36.2352666Z 2022-03-17T20:28:36.2352888Z Running tests... 2022-03-17T20:28:36.2353338Z ---------------------------------------------------------------------- 2022-03-17T20:28:36.2377197Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:28:36.2377461Z 2022-03-17T20:28:36.2377823Z ---------------------------------------------------------------------- 2022-03-17T20:28:36.2378084Z Ran 1 test in 0.002s 2022-03-17T20:28:36.2378260Z 2022-03-17T20:28:36.2378373Z OK (skipped=1) 2022-03-17T20:28:36.2378530Z 2022-03-17T20:28:36.2378657Z Generating XML reports... 2022-03-17T20:28:36.2423264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202836.xml 2022-03-17T20:28:37.3417106Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:37.3434014Z 2022-03-17T20:28:37.3434611Z Running tests... 2022-03-17T20:28:37.3435115Z ---------------------------------------------------------------------- 2022-03-17T20:28:37.3462597Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:28:37.3463046Z 2022-03-17T20:28:37.3463953Z ---------------------------------------------------------------------- 2022-03-17T20:28:37.3464445Z Ran 1 test in 0.003s 2022-03-17T20:28:37.3464940Z 2022-03-17T20:28:37.3465048Z OK (skipped=1) 2022-03-17T20:28:37.3465217Z 2022-03-17T20:28:37.3465327Z Generating XML reports... 2022-03-17T20:28:37.3509117Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202837.xml 2022-03-17T20:28:38.5004092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:38.5019502Z 2022-03-17T20:28:38.5019745Z Running tests... 2022-03-17T20:28:38.5020507Z ---------------------------------------------------------------------- 2022-03-17T20:28:38.5045435Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:28:38.5045973Z 2022-03-17T20:28:38.5046269Z ---------------------------------------------------------------------- 2022-03-17T20:28:38.5046584Z Ran 1 test in 0.003s 2022-03-17T20:28:38.5046746Z 2022-03-17T20:28:38.5046854Z OK (skipped=1) 2022-03-17T20:28:38.5047010Z 2022-03-17T20:28:38.5047154Z Generating XML reports... 2022-03-17T20:28:38.5092121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202838.xml 2022-03-17T20:28:39.6520062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:39.6535420Z 2022-03-17T20:28:39.6535659Z Running tests... 2022-03-17T20:28:39.6536154Z ---------------------------------------------------------------------- 2022-03-17T20:28:41.1887604Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:41.2315125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127112 2022-03-17T20:28:41.2444243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127113 2022-03-17T20:28:42.2867885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:42.2894878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:42.2895669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:42.2969604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:42.2976850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:42.3907207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:42.5490518Z ok (2.895s) 2022-03-17T20:28:42.5490698Z 2022-03-17T20:28:42.5491059Z ---------------------------------------------------------------------- 2022-03-17T20:28:42.5491396Z Ran 1 test in 2.896s 2022-03-17T20:28:42.5491569Z 2022-03-17T20:28:42.5491661Z OK 2022-03-17T20:28:42.5491778Z 2022-03-17T20:28:42.5491916Z Generating XML reports... 2022-03-17T20:28:42.5537154Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202839.xml 2022-03-17T20:28:43.8381304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:43.8396939Z 2022-03-17T20:28:43.8397163Z Running tests... 2022-03-17T20:28:43.8397843Z ---------------------------------------------------------------------- 2022-03-17T20:28:45.3884778Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:45.4309473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127225 2022-03-17T20:28:45.4435109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127226 2022-03-17T20:28:46.5269422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:46.5509235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:46.5510753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:46.5572618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:46.5579803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:46.6521419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:46.8482898Z ok (3.008s) 2022-03-17T20:28:46.8483092Z 2022-03-17T20:28:46.8483489Z ---------------------------------------------------------------------- 2022-03-17T20:28:46.8483825Z Ran 1 test in 3.009s 2022-03-17T20:28:46.8483993Z 2022-03-17T20:28:46.8484071Z OK 2022-03-17T20:28:46.8484207Z 2022-03-17T20:28:46.8484346Z Generating XML reports... 2022-03-17T20:28:46.8530690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202843.xml 2022-03-17T20:28:48.1304875Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:48.1321474Z 2022-03-17T20:28:48.1321931Z Running tests... 2022-03-17T20:28:48.1322427Z ---------------------------------------------------------------------- 2022-03-17T20:28:48.1344111Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:28:48.1344360Z 2022-03-17T20:28:48.1344973Z ---------------------------------------------------------------------- 2022-03-17T20:28:48.1345318Z Ran 1 test in 0.002s 2022-03-17T20:28:48.1345485Z 2022-03-17T20:28:48.1345616Z OK (skipped=1) 2022-03-17T20:28:48.1345753Z 2022-03-17T20:28:48.1345882Z Generating XML reports... 2022-03-17T20:28:48.1389926Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202848.xml 2022-03-17T20:28:49.2774318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:49.2789827Z 2022-03-17T20:28:49.2790170Z Running tests... 2022-03-17T20:28:49.2790631Z ---------------------------------------------------------------------- 2022-03-17T20:28:49.2810777Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:28:49.2811106Z 2022-03-17T20:28:49.2811410Z ---------------------------------------------------------------------- 2022-03-17T20:28:49.2811752Z Ran 1 test in 0.002s 2022-03-17T20:28:49.2811888Z 2022-03-17T20:28:49.2812001Z OK (skipped=1) 2022-03-17T20:28:49.2812156Z 2022-03-17T20:28:49.2812279Z Generating XML reports... 2022-03-17T20:28:49.2855553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202849.xml 2022-03-17T20:28:50.4325988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:50.4341117Z 2022-03-17T20:28:50.4341461Z Running tests... 2022-03-17T20:28:50.4341902Z ---------------------------------------------------------------------- 2022-03-17T20:28:51.9600471Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:52.0031939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127408 2022-03-17T20:28:52.0156443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127409 2022-03-17T20:28:53.0910242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:28:53.0990514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:28:53.0991306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:53.1011192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:28:53.1017676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:28:53.2006196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:28:56.2268479Z ok (5.792s) 2022-03-17T20:28:56.2268851Z 2022-03-17T20:28:56.2269642Z ---------------------------------------------------------------------- 2022-03-17T20:28:56.2270097Z Ran 1 test in 5.793s 2022-03-17T20:28:56.2270246Z 2022-03-17T20:28:56.2270340Z OK 2022-03-17T20:28:56.2270477Z 2022-03-17T20:28:56.2270614Z Generating XML reports... 2022-03-17T20:28:56.2315691Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202850.xml 2022-03-17T20:28:57.5337837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:28:57.5353389Z 2022-03-17T20:28:57.5353637Z Running tests... 2022-03-17T20:28:57.5354552Z ---------------------------------------------------------------------- 2022-03-17T20:28:59.0799468Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:28:59.1224672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127535 2022-03-17T20:28:59.1350639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127536 2022-03-17T20:29:00.2013679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:00.2340050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:00.2341130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:00.2418746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:00.2425875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:00.3355318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:03.3449013Z ok (5.809s) 2022-03-17T20:29:03.3449489Z 2022-03-17T20:29:03.3450067Z ---------------------------------------------------------------------- 2022-03-17T20:29:03.3450311Z Ran 1 test in 5.810s 2022-03-17T20:29:03.3450459Z 2022-03-17T20:29:03.3450557Z OK 2022-03-17T20:29:03.3450698Z 2022-03-17T20:29:03.3450898Z Generating XML reports... 2022-03-17T20:29:03.3495307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202857.xml 2022-03-17T20:29:04.6444127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:04.6460099Z 2022-03-17T20:29:04.6460580Z Running tests... 2022-03-17T20:29:04.6461234Z ---------------------------------------------------------------------- 2022-03-17T20:29:06.1866719Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:29:06.2289792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127662 2022-03-17T20:29:06.2415610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127663 2022-03-17T20:29:07.3081389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:07.3621587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:07.3622368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:07.3689110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:07.3695982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:07.4636772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:10.1508645Z ok (5.505s) 2022-03-17T20:29:10.1508945Z 2022-03-17T20:29:10.1509282Z ---------------------------------------------------------------------- 2022-03-17T20:29:10.1509593Z Ran 1 test in 5.505s 2022-03-17T20:29:10.1509759Z 2022-03-17T20:29:10.1509867Z OK 2022-03-17T20:29:10.1509994Z 2022-03-17T20:29:10.1510132Z Generating XML reports... 2022-03-17T20:29:10.1553827Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202904.xml 2022-03-17T20:29:11.4524532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:11.4540557Z 2022-03-17T20:29:11.4541052Z Running tests... 2022-03-17T20:29:11.4541574Z ---------------------------------------------------------------------- 2022-03-17T20:29:13.0162391Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:29:13.0586381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 127785 2022-03-17T20:29:13.0717343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 127786 2022-03-17T20:29:14.1008153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:14.1440241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:14.1441031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:14.1514433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:14.1520975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:14.2454536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:15.8504537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:29:15.8505089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:29:15.8519865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:29:15.8520355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:29:16.2796750Z ok (4.825s) 2022-03-17T20:29:16.2797025Z 2022-03-17T20:29:16.2797443Z ---------------------------------------------------------------------- 2022-03-17T20:29:16.2797787Z Ran 1 test in 4.826s 2022-03-17T20:29:16.2797958Z 2022-03-17T20:29:16.2798032Z OK 2022-03-17T20:29:16.2798174Z 2022-03-17T20:29:16.2798319Z Generating XML reports... 2022-03-17T20:29:16.2845202Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202911.xml 2022-03-17T20:29:17.5770974Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:17.5787059Z 2022-03-17T20:29:17.5787292Z Running tests... 2022-03-17T20:29:17.5787808Z ---------------------------------------------------------------------- 2022-03-17T20:29:17.5809204Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:17.5809459Z 2022-03-17T20:29:17.5809804Z ---------------------------------------------------------------------- 2022-03-17T20:29:17.5810140Z Ran 1 test in 0.002s 2022-03-17T20:29:17.5810284Z 2022-03-17T20:29:17.5810396Z OK (skipped=1) 2022-03-17T20:29:17.5810488Z 2022-03-17T20:29:17.5810619Z Generating XML reports... 2022-03-17T20:29:17.5863217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202917.xml 2022-03-17T20:29:18.7249281Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:18.7265647Z 2022-03-17T20:29:18.7265777Z Running tests... 2022-03-17T20:29:18.7266807Z ---------------------------------------------------------------------- 2022-03-17T20:29:18.7292011Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:18.7292302Z 2022-03-17T20:29:18.7292625Z ---------------------------------------------------------------------- 2022-03-17T20:29:18.7292945Z Ran 1 test in 0.003s 2022-03-17T20:29:18.7293110Z 2022-03-17T20:29:18.7293221Z OK (skipped=1) 2022-03-17T20:29:18.7293360Z 2022-03-17T20:29:18.7293489Z Generating XML reports... 2022-03-17T20:29:18.7345566Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202918.xml 2022-03-17T20:29:19.8720160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:19.8735526Z 2022-03-17T20:29:19.8735670Z Running tests... 2022-03-17T20:29:19.8736399Z ---------------------------------------------------------------------- 2022-03-17T20:29:19.8763795Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:29:19.8764171Z 2022-03-17T20:29:19.8764413Z ---------------------------------------------------------------------- 2022-03-17T20:29:19.8764747Z Ran 1 test in 0.003s 2022-03-17T20:29:19.8764938Z 2022-03-17T20:29:19.8765042Z OK (skipped=1) 2022-03-17T20:29:19.8765197Z 2022-03-17T20:29:19.8765321Z Generating XML reports... 2022-03-17T20:29:19.8816693Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202919.xml 2022-03-17T20:29:21.0318605Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:21.0338900Z 2022-03-17T20:29:21.0339052Z Running tests... 2022-03-17T20:29:21.0339796Z ---------------------------------------------------------------------- 2022-03-17T20:29:21.0366611Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:29:21.0367021Z 2022-03-17T20:29:21.0367358Z ---------------------------------------------------------------------- 2022-03-17T20:29:21.0367966Z Ran 1 test in 0.003s 2022-03-17T20:29:21.0368175Z 2022-03-17T20:29:21.0368284Z OK (skipped=1) 2022-03-17T20:29:21.0368330Z 2022-03-17T20:29:21.0368455Z Generating XML reports... 2022-03-17T20:29:21.0416927Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202921.xml 2022-03-17T20:29:22.1896882Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:22.1913514Z 2022-03-17T20:29:22.1913723Z Running tests... 2022-03-17T20:29:22.1914248Z ---------------------------------------------------------------------- 2022-03-17T20:29:22.1939607Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:22.1939901Z 2022-03-17T20:29:22.1940509Z ---------------------------------------------------------------------- 2022-03-17T20:29:22.1940847Z Ran 1 test in 0.003s 2022-03-17T20:29:22.1941012Z 2022-03-17T20:29:22.1941417Z OK (skipped=1) 2022-03-17T20:29:22.1941577Z 2022-03-17T20:29:22.1941685Z Generating XML reports... 2022-03-17T20:29:22.1993187Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202922.xml 2022-03-17T20:29:23.2861777Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:23.2877569Z 2022-03-17T20:29:23.2877825Z Running tests... 2022-03-17T20:29:23.2878242Z ---------------------------------------------------------------------- 2022-03-17T20:29:23.2903795Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:23.2904064Z 2022-03-17T20:29:23.2904621Z ---------------------------------------------------------------------- 2022-03-17T20:29:23.2904950Z Ran 1 test in 0.003s 2022-03-17T20:29:23.2905121Z 2022-03-17T20:29:23.2905232Z OK (skipped=1) 2022-03-17T20:29:23.2905390Z 2022-03-17T20:29:23.2905497Z Generating XML reports... 2022-03-17T20:29:23.2948793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202923.xml 2022-03-17T20:29:24.4426641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:24.4441745Z 2022-03-17T20:29:24.4442239Z Running tests... 2022-03-17T20:29:24.4442729Z ---------------------------------------------------------------------- 2022-03-17T20:29:24.4465352Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:24.4465632Z 2022-03-17T20:29:24.4465915Z ---------------------------------------------------------------------- 2022-03-17T20:29:24.4466226Z Ran 1 test in 0.002s 2022-03-17T20:29:24.4466405Z 2022-03-17T20:29:24.4466512Z OK (skipped=1) 2022-03-17T20:29:24.4466667Z 2022-03-17T20:29:24.4466789Z Generating XML reports... 2022-03-17T20:29:24.4511667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202924.xml 2022-03-17T20:29:25.5475271Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:25.5491268Z 2022-03-17T20:29:25.5491924Z Running tests... 2022-03-17T20:29:25.5492433Z ---------------------------------------------------------------------- 2022-03-17T20:29:25.5514422Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:25.5514726Z 2022-03-17T20:29:25.5514979Z ---------------------------------------------------------------------- 2022-03-17T20:29:25.5515305Z Ran 1 test in 0.002s 2022-03-17T20:29:25.5515460Z 2022-03-17T20:29:25.5515551Z OK (skipped=1) 2022-03-17T20:29:25.5515756Z 2022-03-17T20:29:25.5527490Z Generating XML reports... 2022-03-17T20:29:25.5559870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202925.xml 2022-03-17T20:29:26.6504254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:26.6520032Z 2022-03-17T20:29:26.6520426Z Running tests... 2022-03-17T20:29:26.6520917Z ---------------------------------------------------------------------- 2022-03-17T20:29:26.6541609Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:26.6541862Z 2022-03-17T20:29:26.6542145Z ---------------------------------------------------------------------- 2022-03-17T20:29:26.6542476Z Ran 1 test in 0.002s 2022-03-17T20:29:26.6542620Z 2022-03-17T20:29:26.6542730Z OK (skipped=1) 2022-03-17T20:29:26.6542889Z 2022-03-17T20:29:26.6543014Z Generating XML reports... 2022-03-17T20:29:26.6586868Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202926.xml 2022-03-17T20:29:27.8025911Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:27.8041052Z 2022-03-17T20:29:27.8041210Z Running tests... 2022-03-17T20:29:27.8042085Z ---------------------------------------------------------------------- 2022-03-17T20:29:27.8062990Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:27.8063260Z 2022-03-17T20:29:27.8063625Z ---------------------------------------------------------------------- 2022-03-17T20:29:27.8063936Z Ran 1 test in 0.002s 2022-03-17T20:29:27.8064097Z 2022-03-17T20:29:27.8064220Z OK (skipped=1) 2022-03-17T20:29:27.8064367Z 2022-03-17T20:29:27.8064458Z Generating XML reports... 2022-03-17T20:29:27.8108452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202927.xml 2022-03-17T20:29:28.9506493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:28.9521236Z 2022-03-17T20:29:28.9521440Z Running tests... 2022-03-17T20:29:28.9522175Z ---------------------------------------------------------------------- 2022-03-17T20:29:30.5105546Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:29:30.5537346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128261 2022-03-17T20:29:30.5671927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128262 2022-03-17T20:29:31.6191403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:31.6408465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:31.6409519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:31.6495066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:31.6503090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:31.7423654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:33.8754949Z ok (4.923s) 2022-03-17T20:29:33.8755190Z 2022-03-17T20:29:33.8755606Z ---------------------------------------------------------------------- 2022-03-17T20:29:33.8755896Z Ran 1 test in 4.923s 2022-03-17T20:29:33.8756003Z 2022-03-17T20:29:33.8756094Z OK 2022-03-17T20:29:33.8756223Z 2022-03-17T20:29:33.8756343Z Generating XML reports... 2022-03-17T20:29:33.8801180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202928.xml 2022-03-17T20:29:35.1771350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:35.1786676Z 2022-03-17T20:29:35.1787009Z Running tests... 2022-03-17T20:29:35.1787469Z ---------------------------------------------------------------------- 2022-03-17T20:29:35.1810439Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:35.1810696Z 2022-03-17T20:29:35.1811049Z ---------------------------------------------------------------------- 2022-03-17T20:29:35.1811374Z Ran 1 test in 0.002s 2022-03-17T20:29:35.1811542Z 2022-03-17T20:29:35.1811626Z OK (skipped=1) 2022-03-17T20:29:35.1811732Z 2022-03-17T20:29:35.1811828Z Generating XML reports... 2022-03-17T20:29:35.1856243Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202935.xml 2022-03-17T20:29:36.3261477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:36.3276987Z 2022-03-17T20:29:36.3277369Z Running tests... 2022-03-17T20:29:36.3277755Z ---------------------------------------------------------------------- 2022-03-17T20:29:36.3299971Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:36.3300133Z 2022-03-17T20:29:36.3300495Z ---------------------------------------------------------------------- 2022-03-17T20:29:36.3301066Z Ran 1 test in 0.002s 2022-03-17T20:29:36.3301450Z 2022-03-17T20:29:36.3301567Z OK (skipped=1) 2022-03-17T20:29:36.3301743Z 2022-03-17T20:29:36.3301867Z Generating XML reports... 2022-03-17T20:29:36.3345984Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202936.xml 2022-03-17T20:29:37.4801441Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:37.4816249Z 2022-03-17T20:29:37.4816584Z Running tests... 2022-03-17T20:29:37.4817038Z ---------------------------------------------------------------------- 2022-03-17T20:29:39.0136639Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:29:39.0560165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128457 2022-03-17T20:29:39.0685328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128458 2022-03-17T20:29:40.1112686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:40.1544810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:40.1545640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:40.1619000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:40.1626342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:40.2560533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:42.3769141Z ok (4.895s) 2022-03-17T20:29:42.3769347Z 2022-03-17T20:29:42.3769745Z ---------------------------------------------------------------------- 2022-03-17T20:29:42.3770091Z Ran 1 test in 4.895s 2022-03-17T20:29:42.3770266Z 2022-03-17T20:29:42.3770411Z OK 2022-03-17T20:29:42.3770468Z 2022-03-17T20:29:42.3770607Z Generating XML reports... 2022-03-17T20:29:42.3816199Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202937.xml 2022-03-17T20:29:43.6844627Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:43.6859311Z 2022-03-17T20:29:43.6859569Z Running tests... 2022-03-17T20:29:43.6860012Z ---------------------------------------------------------------------- 2022-03-17T20:29:45.2278752Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:29:45.2704052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128583 2022-03-17T20:29:45.2830168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128584 2022-03-17T20:29:46.3126019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:46.3216062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:46.3216872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:46.3227525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:46.3234390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:46.4231512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:29:48.5926576Z ok (4.906s) 2022-03-17T20:29:48.5926839Z 2022-03-17T20:29:48.5927391Z ---------------------------------------------------------------------- 2022-03-17T20:29:48.5927736Z Ran 1 test in 4.907s 2022-03-17T20:29:48.5927906Z 2022-03-17T20:29:48.5927978Z OK 2022-03-17T20:29:48.5928113Z 2022-03-17T20:29:48.5928247Z Generating XML reports... 2022-03-17T20:29:48.5972699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202943.xml 2022-03-17T20:29:49.9097388Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:49.9113455Z 2022-03-17T20:29:49.9113607Z Running tests... 2022-03-17T20:29:49.9114080Z ---------------------------------------------------------------------- 2022-03-17T20:29:49.9136215Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:49.9136507Z 2022-03-17T20:29:49.9136748Z ---------------------------------------------------------------------- 2022-03-17T20:29:49.9137406Z Ran 1 test in 0.002s 2022-03-17T20:29:49.9137583Z 2022-03-17T20:29:49.9137695Z OK (skipped=1) 2022-03-17T20:29:49.9137791Z 2022-03-17T20:29:49.9137889Z Generating XML reports... 2022-03-17T20:29:49.9181660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202949.xml 2022-03-17T20:29:51.0521196Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:51.0536495Z 2022-03-17T20:29:51.0536789Z Running tests... 2022-03-17T20:29:51.0537243Z ---------------------------------------------------------------------- 2022-03-17T20:29:51.0557161Z test_scatter (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:51.0557414Z 2022-03-17T20:29:51.0557683Z ---------------------------------------------------------------------- 2022-03-17T20:29:51.0558014Z Ran 1 test in 0.002s 2022-03-17T20:29:51.0558178Z 2022-03-17T20:29:51.0558292Z OK (skipped=1) 2022-03-17T20:29:51.0558435Z 2022-03-17T20:29:51.0558582Z Generating XML reports... 2022-03-17T20:29:51.0602280Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202951.xml 2022-03-17T20:29:52.2052310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:52.2068053Z 2022-03-17T20:29:52.2068287Z Running tests... 2022-03-17T20:29:52.2068734Z ---------------------------------------------------------------------- 2022-03-17T20:29:52.2096381Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:29:52.2096637Z 2022-03-17T20:29:52.2096956Z ---------------------------------------------------------------------- 2022-03-17T20:29:52.2097292Z Ran 1 test in 0.003s 2022-03-17T20:29:52.2097456Z 2022-03-17T20:29:52.2097575Z OK (skipped=1) 2022-03-17T20:29:52.2097713Z 2022-03-17T20:29:52.2097839Z Generating XML reports... 2022-03-17T20:29:52.2141964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202952.xml 2022-03-17T20:29:53.3558191Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:53.3574234Z 2022-03-17T20:29:53.3574344Z Running tests... 2022-03-17T20:29:53.3574856Z ---------------------------------------------------------------------- 2022-03-17T20:29:53.3595366Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:29:53.3595621Z 2022-03-17T20:29:53.3595920Z ---------------------------------------------------------------------- 2022-03-17T20:29:53.3596251Z Ran 1 test in 0.002s 2022-03-17T20:29:53.3596403Z 2022-03-17T20:29:53.3596505Z OK (skipped=1) 2022-03-17T20:29:53.3596658Z 2022-03-17T20:29:53.3596734Z Generating XML reports... 2022-03-17T20:29:53.3641659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202953.xml 2022-03-17T20:29:54.5059080Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:29:54.5074845Z 2022-03-17T20:29:54.5075003Z Running tests... 2022-03-17T20:29:54.5075849Z ---------------------------------------------------------------------- 2022-03-17T20:29:56.0745704Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:29:56.1171895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128849 2022-03-17T20:29:56.1297549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128850 2022-03-17T20:29:57.1895650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:29:57.2117823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:29:57.2118623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:57.2200486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:29:57.2207625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:29:57.3133458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:01.2408708Z ok (6.733s) 2022-03-17T20:30:01.2408905Z 2022-03-17T20:30:01.2409386Z ---------------------------------------------------------------------- 2022-03-17T20:30:01.2409732Z Ran 1 test in 6.733s 2022-03-17T20:30:01.2409896Z 2022-03-17T20:30:01.2409945Z OK 2022-03-17T20:30:01.2410078Z 2022-03-17T20:30:01.2410199Z Generating XML reports... 2022-03-17T20:30:01.2456189Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202954.xml 2022-03-17T20:30:02.5753926Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:02.5770022Z 2022-03-17T20:30:02.5770523Z Running tests... 2022-03-17T20:30:02.5771019Z ---------------------------------------------------------------------- 2022-03-17T20:30:04.1149627Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:30:04.1582208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 128977 2022-03-17T20:30:04.1709637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 128978 2022-03-17T20:30:05.2066943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:05.2113236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:05.2114051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:05.2170234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:05.2177109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:05.3128698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:09.2825486Z ok (6.705s) 2022-03-17T20:30:09.2825708Z 2022-03-17T20:30:09.2826084Z ---------------------------------------------------------------------- 2022-03-17T20:30:09.2826445Z Ran 1 test in 6.706s 2022-03-17T20:30:09.2826616Z 2022-03-17T20:30:09.2826713Z OK 2022-03-17T20:30:09.2826831Z 2022-03-17T20:30:09.2826917Z Generating XML reports... 2022-03-17T20:30:09.2872708Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203002.xml 2022-03-17T20:30:10.5788594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:10.5802312Z 2022-03-17T20:30:10.5802435Z Running tests... 2022-03-17T20:30:10.5803307Z ---------------------------------------------------------------------- 2022-03-17T20:30:10.5823550Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:10.5823888Z 2022-03-17T20:30:10.5824198Z ---------------------------------------------------------------------- 2022-03-17T20:30:10.5824820Z Ran 1 test in 0.002s 2022-03-17T20:30:10.5824911Z 2022-03-17T20:30:10.5825021Z OK (skipped=1) 2022-03-17T20:30:10.5825177Z 2022-03-17T20:30:10.5825292Z Generating XML reports... 2022-03-17T20:30:10.5869766Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203010.xml 2022-03-17T20:30:11.7338076Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:11.7353999Z 2022-03-17T20:30:11.7354140Z Running tests... 2022-03-17T20:30:11.7354730Z ---------------------------------------------------------------------- 2022-03-17T20:30:11.7375458Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:11.7375749Z 2022-03-17T20:30:11.7376040Z ---------------------------------------------------------------------- 2022-03-17T20:30:11.7376351Z Ran 1 test in 0.002s 2022-03-17T20:30:11.7376513Z 2022-03-17T20:30:11.7376631Z OK (skipped=1) 2022-03-17T20:30:11.7376789Z 2022-03-17T20:30:11.7376895Z Generating XML reports... 2022-03-17T20:30:11.7421153Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203011.xml 2022-03-17T20:30:12.8737071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:12.8752195Z 2022-03-17T20:30:12.8752421Z Running tests... 2022-03-17T20:30:12.8753086Z ---------------------------------------------------------------------- 2022-03-17T20:30:12.8778091Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:12.8778612Z 2022-03-17T20:30:12.8778915Z ---------------------------------------------------------------------- 2022-03-17T20:30:12.8779246Z Ran 1 test in 0.003s 2022-03-17T20:30:12.8779353Z 2022-03-17T20:30:12.8779462Z OK (skipped=1) 2022-03-17T20:30:12.8779652Z 2022-03-17T20:30:12.8779786Z Generating XML reports... 2022-03-17T20:30:12.8822779Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203012.xml 2022-03-17T20:30:14.0194663Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:14.0210374Z 2022-03-17T20:30:14.0210658Z Running tests... 2022-03-17T20:30:14.0211112Z ---------------------------------------------------------------------- 2022-03-17T20:30:14.0230535Z test_send_recv (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:14.0230986Z 2022-03-17T20:30:14.0231427Z ---------------------------------------------------------------------- 2022-03-17T20:30:14.0231802Z Ran 1 test in 0.002s 2022-03-17T20:30:14.0231994Z 2022-03-17T20:30:14.0232111Z OK (skipped=1) 2022-03-17T20:30:14.0232252Z 2022-03-17T20:30:14.0232378Z Generating XML reports... 2022-03-17T20:30:14.0277305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203014.xml 2022-03-17T20:30:15.1677097Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:15.1692638Z 2022-03-17T20:30:15.1693147Z Running tests... 2022-03-17T20:30:15.1693584Z ---------------------------------------------------------------------- 2022-03-17T20:30:15.1713208Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:15.1713501Z 2022-03-17T20:30:15.1713800Z ---------------------------------------------------------------------- 2022-03-17T20:30:15.1714104Z Ran 1 test in 0.002s 2022-03-17T20:30:15.1714257Z 2022-03-17T20:30:15.1714358Z OK (skipped=1) 2022-03-17T20:30:15.1714513Z 2022-03-17T20:30:15.1714666Z Generating XML reports... 2022-03-17T20:30:15.1759107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203015.xml 2022-03-17T20:30:16.3249342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:16.3265604Z 2022-03-17T20:30:16.3265758Z Running tests... 2022-03-17T20:30:16.3266382Z ---------------------------------------------------------------------- 2022-03-17T20:30:16.3286441Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:16.3286743Z 2022-03-17T20:30:16.3287027Z ---------------------------------------------------------------------- 2022-03-17T20:30:16.3287360Z Ran 1 test in 0.002s 2022-03-17T20:30:16.3287503Z 2022-03-17T20:30:16.3287615Z OK (skipped=1) 2022-03-17T20:30:16.3287775Z 2022-03-17T20:30:16.3287904Z Generating XML reports... 2022-03-17T20:30:16.3332468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203016.xml 2022-03-17T20:30:17.4749431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:17.4765246Z 2022-03-17T20:30:17.4765375Z Running tests... 2022-03-17T20:30:17.4766116Z ---------------------------------------------------------------------- 2022-03-17T20:30:17.4787084Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:17.4787376Z 2022-03-17T20:30:17.4787646Z ---------------------------------------------------------------------- 2022-03-17T20:30:17.4787989Z Ran 1 test in 0.002s 2022-03-17T20:30:17.4788153Z 2022-03-17T20:30:17.4788480Z OK (skipped=1) 2022-03-17T20:30:17.4788620Z 2022-03-17T20:30:17.4789136Z Generating XML reports... 2022-03-17T20:30:17.4834143Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203017.xml 2022-03-17T20:30:18.6246377Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:18.6262090Z 2022-03-17T20:30:18.6262241Z Running tests... 2022-03-17T20:30:18.6262977Z ---------------------------------------------------------------------- 2022-03-17T20:30:18.6282889Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:18.6283277Z 2022-03-17T20:30:18.6283577Z ---------------------------------------------------------------------- 2022-03-17T20:30:18.6283909Z Ran 1 test in 0.002s 2022-03-17T20:30:18.6284074Z 2022-03-17T20:30:18.6284160Z OK (skipped=1) 2022-03-17T20:30:18.6284309Z 2022-03-17T20:30:18.6284397Z Generating XML reports... 2022-03-17T20:30:18.6328807Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203018.xml 2022-03-17T20:30:19.7766598Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:19.7782229Z 2022-03-17T20:30:19.7782397Z Running tests... 2022-03-17T20:30:19.7782836Z ---------------------------------------------------------------------- 2022-03-17T20:30:21.3262003Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:30:21.3686682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129385 2022-03-17T20:30:21.3812181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129386 2022-03-17T20:30:22.4214892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:22.4590635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:22.4591411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:22.4620588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:22.4628068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:22.5606743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:24.1885692Z ok (4.410s) 2022-03-17T20:30:24.1886101Z 2022-03-17T20:30:24.1886803Z ---------------------------------------------------------------------- 2022-03-17T20:30:24.1887353Z Ran 1 test in 4.410s 2022-03-17T20:30:24.1887648Z 2022-03-17T20:30:24.1887849Z OK 2022-03-17T20:30:24.1888056Z 2022-03-17T20:30:24.1888250Z Generating XML reports... 2022-03-17T20:30:24.1934358Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203019.xml 2022-03-17T20:30:25.4804205Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:25.4819209Z 2022-03-17T20:30:25.4819357Z Running tests... 2022-03-17T20:30:25.4820657Z ---------------------------------------------------------------------- 2022-03-17T20:30:27.0146848Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:30:27.0570178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129507 2022-03-17T20:30:27.0696467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129508 2022-03-17T20:30:28.1407960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:28.1488586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:28.1489761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:28.1511082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:28.1517991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:28.2504301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:30.3779011Z ok (4.896s) 2022-03-17T20:30:30.3779238Z 2022-03-17T20:30:30.3779619Z ---------------------------------------------------------------------- 2022-03-17T20:30:30.3779964Z Ran 1 test in 4.896s 2022-03-17T20:30:30.3780131Z 2022-03-17T20:30:30.3780498Z OK 2022-03-17T20:30:30.3780636Z 2022-03-17T20:30:30.3780771Z Generating XML reports... 2022-03-17T20:30:30.3824783Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203025.xml 2022-03-17T20:30:31.6763978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:31.6779412Z 2022-03-17T20:30:31.6779570Z Running tests... 2022-03-17T20:30:31.6780260Z ---------------------------------------------------------------------- 2022-03-17T20:30:33.2073882Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:30:33.2498874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129633 2022-03-17T20:30:33.2622852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129634 2022-03-17T20:30:34.3055891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:34.3424886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:34.3425653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:34.3461259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:34.3468480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:34.4439544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:36.5705972Z ok (4.892s) 2022-03-17T20:30:36.5706309Z 2022-03-17T20:30:36.5706880Z ---------------------------------------------------------------------- 2022-03-17T20:30:36.5707475Z Ran 1 test in 4.893s 2022-03-17T20:30:36.5707647Z 2022-03-17T20:30:36.5707744Z OK 2022-03-17T20:30:36.5707860Z 2022-03-17T20:30:36.5708000Z Generating XML reports... 2022-03-17T20:30:36.5753568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203031.xml 2022-03-17T20:30:37.8781293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:37.8796573Z 2022-03-17T20:30:37.8796735Z Running tests... 2022-03-17T20:30:37.8797229Z ---------------------------------------------------------------------- 2022-03-17T20:30:37.8818751Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:37.8819146Z 2022-03-17T20:30:37.8819752Z ---------------------------------------------------------------------- 2022-03-17T20:30:37.8820111Z Ran 1 test in 0.002s 2022-03-17T20:30:37.8820472Z 2022-03-17T20:30:37.8820622Z OK (skipped=1) 2022-03-17T20:30:37.8820748Z 2022-03-17T20:30:37.8820879Z Generating XML reports... 2022-03-17T20:30:37.8862750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203037.xml 2022-03-17T20:30:39.0263184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:39.0278757Z 2022-03-17T20:30:39.0279027Z Running tests... 2022-03-17T20:30:39.0279450Z ---------------------------------------------------------------------- 2022-03-17T20:30:39.0298355Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:39.0298569Z 2022-03-17T20:30:39.0298884Z ---------------------------------------------------------------------- 2022-03-17T20:30:39.0299212Z Ran 1 test in 0.002s 2022-03-17T20:30:39.0299390Z 2022-03-17T20:30:39.0299520Z OK (skipped=1) 2022-03-17T20:30:39.0299624Z 2022-03-17T20:30:39.0299698Z Generating XML reports... 2022-03-17T20:30:39.0343617Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203039.xml 2022-03-17T20:30:40.1850746Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:40.1866384Z 2022-03-17T20:30:40.1866617Z Running tests... 2022-03-17T20:30:40.1867070Z ---------------------------------------------------------------------- 2022-03-17T20:30:40.1886373Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:40.1886667Z 2022-03-17T20:30:40.1886930Z ---------------------------------------------------------------------- 2022-03-17T20:30:40.1887275Z Ran 1 test in 0.002s 2022-03-17T20:30:40.1887461Z 2022-03-17T20:30:40.1887556Z OK (skipped=1) 2022-03-17T20:30:40.1887717Z 2022-03-17T20:30:40.1887851Z Generating XML reports... 2022-03-17T20:30:40.1932174Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203040.xml 2022-03-17T20:30:41.3403014Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:41.3418244Z 2022-03-17T20:30:41.3418659Z Running tests... 2022-03-17T20:30:41.3419124Z ---------------------------------------------------------------------- 2022-03-17T20:30:41.3439728Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:41.3440162Z 2022-03-17T20:30:41.3440846Z ---------------------------------------------------------------------- 2022-03-17T20:30:41.3441451Z Ran 1 test in 0.002s 2022-03-17T20:30:41.3441625Z 2022-03-17T20:30:41.3441735Z OK (skipped=1) 2022-03-17T20:30:41.3441926Z 2022-03-17T20:30:41.3442034Z Generating XML reports... 2022-03-17T20:30:41.3485973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203041.xml 2022-03-17T20:30:42.5033141Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:42.5049126Z 2022-03-17T20:30:42.5049438Z Running tests... 2022-03-17T20:30:42.5049924Z ---------------------------------------------------------------------- 2022-03-17T20:30:42.5069574Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:42.5069891Z 2022-03-17T20:30:42.5070682Z ---------------------------------------------------------------------- 2022-03-17T20:30:42.5071059Z Ran 1 test in 0.002s 2022-03-17T20:30:42.5071209Z 2022-03-17T20:30:42.5071326Z OK (skipped=1) 2022-03-17T20:30:42.5071495Z 2022-03-17T20:30:42.5071627Z Generating XML reports... 2022-03-17T20:30:42.5116467Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203042.xml 2022-03-17T20:30:43.6091811Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:43.6108270Z 2022-03-17T20:30:43.6108526Z Running tests... 2022-03-17T20:30:43.6109048Z ---------------------------------------------------------------------- 2022-03-17T20:30:43.6129065Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:43.6129333Z 2022-03-17T20:30:43.6130378Z ---------------------------------------------------------------------- 2022-03-17T20:30:43.6131046Z Ran 1 test in 0.002s 2022-03-17T20:30:43.6131356Z 2022-03-17T20:30:43.6131563Z OK (skipped=1) 2022-03-17T20:30:43.6131878Z 2022-03-17T20:30:43.6132091Z Generating XML reports... 2022-03-17T20:30:43.6178070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203043.xml 2022-03-17T20:30:44.7577160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:44.7592331Z 2022-03-17T20:30:44.7592678Z Running tests... 2022-03-17T20:30:44.7593103Z ---------------------------------------------------------------------- 2022-03-17T20:30:46.3099366Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:30:46.3520108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 129969 2022-03-17T20:30:46.3645186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 129970 2022-03-17T20:30:47.4396163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:47.4529376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:47.4530466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:47.4598729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:47.4605432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:47.5544680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:50.5740118Z ok (5.814s) 2022-03-17T20:30:50.5740643Z 2022-03-17T20:30:50.5741208Z ---------------------------------------------------------------------- 2022-03-17T20:30:50.5741537Z Ran 1 test in 5.815s 2022-03-17T20:30:50.5741706Z 2022-03-17T20:30:50.5741799Z OK 2022-03-17T20:30:50.5741940Z 2022-03-17T20:30:50.5742055Z Generating XML reports... 2022-03-17T20:30:50.5786960Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203044.xml 2022-03-17T20:30:51.8709392Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:51.8724058Z 2022-03-17T20:30:51.8724313Z Running tests... 2022-03-17T20:30:51.8724734Z ---------------------------------------------------------------------- 2022-03-17T20:30:51.8748850Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:30:51.8749397Z 2022-03-17T20:30:51.8749699Z ---------------------------------------------------------------------- 2022-03-17T20:30:51.8750044Z Ran 1 test in 0.002s 2022-03-17T20:30:51.8750131Z 2022-03-17T20:30:51.8750240Z OK (skipped=1) 2022-03-17T20:30:51.8750397Z 2022-03-17T20:30:51.8750505Z Generating XML reports... 2022-03-17T20:30:51.8793622Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203051.xml 2022-03-17T20:30:53.0151457Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:53.0166289Z 2022-03-17T20:30:53.0166676Z Running tests... 2022-03-17T20:30:53.0167686Z ---------------------------------------------------------------------- 2022-03-17T20:30:54.5701916Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:30:54.6124860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130131 2022-03-17T20:30:54.6250267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130132 2022-03-17T20:30:55.6588456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:30:55.6993366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:30:55.6994148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:55.6994849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:30:55.7001868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:30:55.7002334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:30:58.4356083Z ok (5.419s) 2022-03-17T20:30:58.4356320Z 2022-03-17T20:30:58.4356695Z ---------------------------------------------------------------------- 2022-03-17T20:30:58.4357034Z Ran 1 test in 5.419s 2022-03-17T20:30:58.4357204Z 2022-03-17T20:30:58.4357299Z OK 2022-03-17T20:30:58.4357414Z 2022-03-17T20:30:58.4357550Z Generating XML reports... 2022-03-17T20:30:58.4402351Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203053.xml 2022-03-17T20:30:59.7342031Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:30:59.7357716Z 2022-03-17T20:30:59.7358063Z Running tests... 2022-03-17T20:30:59.7358573Z ---------------------------------------------------------------------- 2022-03-17T20:31:01.2845249Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:31:01.3294300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130254 2022-03-17T20:31:01.3429629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130255 2022-03-17T20:31:02.4152568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:02.4479394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:02.4480198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:02.4557693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:02.4564485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:02.5494315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:05.1339877Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:31:05.1421926Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:31:05.5529099Z ok (5.817s) 2022-03-17T20:31:05.5529252Z 2022-03-17T20:31:05.5529704Z ---------------------------------------------------------------------- 2022-03-17T20:31:05.5530039Z Ran 1 test in 5.817s 2022-03-17T20:31:05.5530217Z 2022-03-17T20:31:05.5530217Z OK 2022-03-17T20:31:05.5530327Z 2022-03-17T20:31:05.5530445Z Generating XML reports... 2022-03-17T20:31:05.5575939Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203059.xml 2022-03-17T20:31:06.8677887Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:06.8693099Z 2022-03-17T20:31:06.8693480Z Running tests... 2022-03-17T20:31:06.8694036Z ---------------------------------------------------------------------- 2022-03-17T20:31:08.4183058Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:31:08.4611793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130381 2022-03-17T20:31:08.4735995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130382 2022-03-17T20:31:09.5630760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:09.5735974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:09.5736859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:09.5834287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:09.5841544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:09.6749378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:09.6859497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:31:09.6860742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:31:09.6861913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:31:09.6865505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:31:09.6962099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:31:09.6965200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:31:09.6966526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:31:09.6967803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:31:16.8891319Z ok (10.019s) 2022-03-17T20:31:16.8891577Z 2022-03-17T20:31:16.8891941Z ---------------------------------------------------------------------- 2022-03-17T20:31:16.8892327Z Ran 1 test in 10.020s 2022-03-17T20:31:16.8892444Z 2022-03-17T20:31:16.8892613Z OK 2022-03-17T20:31:16.8892751Z 2022-03-17T20:31:16.8892787Z Generating XML reports... 2022-03-17T20:31:16.8938634Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203106.xml 2022-03-17T20:31:18.2058164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:18.2075318Z 2022-03-17T20:31:18.2075937Z Running tests... 2022-03-17T20:31:18.2076392Z ---------------------------------------------------------------------- 2022-03-17T20:31:19.7685057Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:31:19.8129126Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130511 2022-03-17T20:31:19.8261208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130512 2022-03-17T20:31:20.8757088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:20.9289986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:20.9290812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:20.9365474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:20.9372154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:21.0301822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:21.0488938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:31:21.0489594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:31:21.0490318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:31:21.0491012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:31:21.0493099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:31:21.0593604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:31:21.0594337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:31:21.0595240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:31:28.1435571Z ok (9.936s) 2022-03-17T20:31:28.1435777Z 2022-03-17T20:31:28.1436207Z ---------------------------------------------------------------------- 2022-03-17T20:31:28.1436534Z Ran 1 test in 9.936s 2022-03-17T20:31:28.1436661Z 2022-03-17T20:31:28.1436811Z OK 2022-03-17T20:31:28.1436923Z 2022-03-17T20:31:28.1437009Z Generating XML reports... 2022-03-17T20:31:28.1483706Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203118.xml 2022-03-17T20:31:28.4988923Z Running distributed tests for the nccl backend with file init_method 2022-03-17T20:31:28.4989846Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:31:28.498612] 2022-03-17T20:31:29.5599880Z 2022-03-17T20:31:29.5640834Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_get_bucket_sizes>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_params_and_buffers>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-03-17T20:31:29.5674622Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5675140Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5675541Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5675952Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5676379Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5676846Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5677320Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5677789Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5678297Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5678815Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5679365Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5679884Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5680387Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5680897Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5681370Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5681832Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5682294Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5682701Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5683121Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5683561Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5684134Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5684597Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5684984Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5685375Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5685797Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5686198Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5686698Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5687112Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5687523Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5687889Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5688279Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5688674Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5689041Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5689425Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5689808Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5690222Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5690630Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5691040Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5691475Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5691896Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5692337Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5692757Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5693157Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5693580Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5693988Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5694393Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5694827Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5695245Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5695654Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5696041Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5696461Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5696878Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5697262Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5697668Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5698059Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5698453Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5698840Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5699218Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5699611Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5699970Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5700864Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5701354Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5701739Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5702139Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5702510Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5702886Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5703246Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5703635Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5704022Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5704460Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5704876Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5705238Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5705606Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5705984Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5706349Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5706739Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5707115Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5707502Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5707881Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5708263Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5708690Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5709102Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5709538Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5709991Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5710424Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5710864Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5711283Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5711709Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5712137Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5712556Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5712997Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5713430Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5713883Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5714334Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5714759Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5715174Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5715542Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5715915Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5716275Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5716619Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5716998Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5717371Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5717753Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5718180Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5718573Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5718973Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5719344Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5719736Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5720115Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5720537Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5720945Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5721383Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5721797Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5722184Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5722595Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5722999Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5723364Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5723730Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5724094Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5724476Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5724858Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5725229Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5725675Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5726161Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5726603Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5727013Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5727404Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5727828Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5728251Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5728707Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5729139Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5729539Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5729973Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5730360Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5730731Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5731117Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5731497Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5731892Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5732281Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5732710Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5733145Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5733539Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5733989Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5734465Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5735013Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5735681Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5736255Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5736839Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5737478Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5738058Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5738646Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5739238Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5739760Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5740708Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5741155Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5741543Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5741940Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5742326Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5742719Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5743120Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5743554Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5744009Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5744452Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5744863Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5745231Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5745634Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5746071Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5746480Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5746890Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5747285Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5747704Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5748127Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5748523Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5748933Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5749329Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5749740Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5750123Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5750538Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5750985Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5751488Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5751867Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5752242Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5752657Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5753070Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5753429Z test_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5753788Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5754137Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5754572Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5754959Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5755309Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5755690Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5756054Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5756410Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5756767Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5757123Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5757513Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5757881Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5758245Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5758568Z test_isend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5758948Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5759344Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5759735Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5760184Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5760618Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5761026Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5761439Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5761849Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5762267Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5762665Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5763077Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5763487Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5763867Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5764325Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5764703Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5765097Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5765556Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5766022Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5766488Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5766912Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5767368Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5767817Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5768224Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5768705Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5769097Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5769504Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5769892Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5770271Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5770670Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5771035Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5771460Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5771855Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5772219Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5772585Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5772930Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5773299Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5773659Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5774024Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5774389Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5774751Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5775131Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5775480Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5775846Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5776218Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5776571Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5776948Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5777321Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5777694Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5778072Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5778426Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5778795Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5779193Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5779630Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5780032Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5780948Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5781347Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5781750Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5782161Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5782533Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5782937Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5783364Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5783754Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5784149Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5784533Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5784931Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5785310Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5785703Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5786143Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:29.5786654Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:31:30.6109070Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:30.6123562Z 2022-03-17T20:31:30.6124017Z Running tests... 2022-03-17T20:31:30.6124538Z ---------------------------------------------------------------------- 2022-03-17T20:31:32.1448614Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:31:32.1872425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130676 2022-03-17T20:31:32.2001095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130677 2022-03-17T20:31:33.2789572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:33.2980486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:33.2981978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:33.2991807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:33.2998578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:33.3996079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:34.6011686Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:31:34.6012668Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:31:34.6727945Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:31:34.6728908Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:31:35.7132505Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:31:35.7133192Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:31:35.7134187Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:31:35.7135087Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:31:35.7270368Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:31:35.7271284Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:31:35.7272441Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:31:35.7273293Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:31:35.7410360Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:31:35.7411236Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:31:35.7413751Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:31:35.7414905Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:31:36.1092160Z ok (5.496s) 2022-03-17T20:31:36.1092526Z 2022-03-17T20:31:36.1092999Z ---------------------------------------------------------------------- 2022-03-17T20:31:36.1093326Z Ran 1 test in 5.497s 2022-03-17T20:31:36.1093495Z 2022-03-17T20:31:36.1093590Z OK 2022-03-17T20:31:36.1093728Z 2022-03-17T20:31:36.1093860Z Generating XML reports... 2022-03-17T20:31:36.1139738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203130.xml 2022-03-17T20:31:37.4201000Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:37.4215786Z 2022-03-17T20:31:37.4216045Z Running tests... 2022-03-17T20:31:37.4216470Z ---------------------------------------------------------------------- 2022-03-17T20:31:37.4256259Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip (0.004s) 2022-03-17T20:31:37.4256593Z 2022-03-17T20:31:37.4256931Z ---------------------------------------------------------------------- 2022-03-17T20:31:37.4257220Z Ran 1 test in 0.004s 2022-03-17T20:31:37.4257388Z 2022-03-17T20:31:37.4257493Z OK (skipped=1) 2022-03-17T20:31:37.4257652Z 2022-03-17T20:31:37.4257730Z Generating XML reports... 2022-03-17T20:31:37.4301387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203137.xml 2022-03-17T20:31:38.5621207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:38.5635976Z 2022-03-17T20:31:38.5636280Z Running tests... 2022-03-17T20:31:38.5636718Z ---------------------------------------------------------------------- 2022-03-17T20:31:40.0823721Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:31:40.1244786Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130834 2022-03-17T20:31:40.1371440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130835 2022-03-17T20:31:41.2008115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:41.2484125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:41.2484906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:41.2514519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:41.2521011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:41.3495693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:41.5419596Z ok (2.978s) 2022-03-17T20:31:41.5419811Z 2022-03-17T20:31:41.5420548Z ---------------------------------------------------------------------- 2022-03-17T20:31:41.5421178Z Ran 1 test in 2.978s 2022-03-17T20:31:41.5421430Z 2022-03-17T20:31:41.5421590Z OK 2022-03-17T20:31:41.5421829Z 2022-03-17T20:31:41.5422029Z Generating XML reports... 2022-03-17T20:31:41.5466870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203138.xml 2022-03-17T20:31:42.8086346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:42.8100789Z 2022-03-17T20:31:42.8101307Z Running tests... 2022-03-17T20:31:42.8101738Z ---------------------------------------------------------------------- 2022-03-17T20:31:44.3411540Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:31:44.3832625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 130947 2022-03-17T20:31:44.3959000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 130948 2022-03-17T20:31:45.4463165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:45.4801240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:45.4802431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:45.4868576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:45.4875372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:45.5816873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:48.2063431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.2063946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.2239479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.2242902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.2316872Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:31:48.2317651Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:31:48.2321694Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:31:48.2322421Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:31:48.2540429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.2544486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.3896911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.3897467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.4070409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.4073647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.5297489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.5300282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.5470927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.5476832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.6746588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.6747525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.8009989Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:31:48.8010761Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:31:48.8011389Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:31:48.8011940Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:31:48.8152556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.8154423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.9554450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:48.9556788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.0957528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.0960134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.2312322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.2312813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.2483877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.2487128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.3758871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.3759389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.5114745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.5115223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.5286850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.5292600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.6519238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.6519763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.6690346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.6695073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.7963787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.7964268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.9370917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:49.9371460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:50.0770468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:50.0771009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:50.2169995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:50.2171105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:50.7085839Z ok (7.898s) 2022-03-17T20:31:50.7086051Z 2022-03-17T20:31:50.7086704Z ---------------------------------------------------------------------- 2022-03-17T20:31:50.7087167Z Ran 1 test in 7.898s 2022-03-17T20:31:50.7087336Z 2022-03-17T20:31:50.7087408Z OK 2022-03-17T20:31:50.7087459Z 2022-03-17T20:31:50.7087590Z Generating XML reports... 2022-03-17T20:31:50.7132524Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203142.xml 2022-03-17T20:31:52.0091003Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:52.0106135Z 2022-03-17T20:31:52.0106378Z Running tests... 2022-03-17T20:31:52.0107048Z ---------------------------------------------------------------------- 2022-03-17T20:31:52.0125469Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:52.0125948Z 2022-03-17T20:31:52.0126472Z ---------------------------------------------------------------------- 2022-03-17T20:31:52.0127044Z Ran 1 test in 0.002s 2022-03-17T20:31:52.0127220Z 2022-03-17T20:31:52.0127338Z OK (skipped=1) 2022-03-17T20:31:52.0127774Z 2022-03-17T20:31:52.0127920Z Generating XML reports... 2022-03-17T20:31:52.0170314Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203152.xml 2022-03-17T20:31:53.1528537Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:53.1543750Z 2022-03-17T20:31:53.1544250Z Running tests... 2022-03-17T20:31:53.1544723Z ---------------------------------------------------------------------- 2022-03-17T20:31:53.1563054Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:31:53.1563414Z 2022-03-17T20:31:53.1563692Z ---------------------------------------------------------------------- 2022-03-17T20:31:53.1564023Z Ran 1 test in 0.002s 2022-03-17T20:31:53.1564182Z 2022-03-17T20:31:53.1564260Z OK (skipped=1) 2022-03-17T20:31:53.1564443Z 2022-03-17T20:31:53.1564570Z Generating XML reports... 2022-03-17T20:31:53.1608329Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203153.xml 2022-03-17T20:31:54.3009674Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:31:54.3024974Z 2022-03-17T20:31:54.3025384Z Running tests... 2022-03-17T20:31:54.3025898Z ---------------------------------------------------------------------- 2022-03-17T20:31:55.8446548Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:31:55.8870367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 372 2022-03-17T20:31:55.9000437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 373 2022-03-17T20:31:56.9364931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:31:56.9390371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:31:56.9391187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:56.9466024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:31:56.9472882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:31:57.0406507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:31:59.9927655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:31:59.9928664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.0131611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.0132552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.0221534Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:00.0223417Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:00.0225373Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:00.0226833Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:00.0430030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.0430982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.0631377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.0632360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.1904841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.1905786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.2106597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.2110241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:00.8109973Z ok (6.508s) 2022-03-17T20:32:00.8110126Z 2022-03-17T20:32:00.8110613Z ---------------------------------------------------------------------- 2022-03-17T20:32:00.8110956Z Ran 1 test in 6.509s 2022-03-17T20:32:00.8111123Z 2022-03-17T20:32:00.8111195Z OK 2022-03-17T20:32:00.8111346Z 2022-03-17T20:32:00.8111473Z Generating XML reports... 2022-03-17T20:32:00.8156559Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203154.xml 2022-03-17T20:32:02.1148449Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:02.1163505Z 2022-03-17T20:32:02.1163796Z Running tests... 2022-03-17T20:32:02.1164246Z ---------------------------------------------------------------------- 2022-03-17T20:32:03.6707274Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:03.7129274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 499 2022-03-17T20:32:03.7254946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 500 2022-03-17T20:32:04.7898963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:04.8283995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:04.8284797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:04.8304176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:04.8311380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:04.9298940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:07.2812900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:07.2813431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:07.2949413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:07.2949890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:07.3011780Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:07.3012755Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:07.3013932Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:07.3014705Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:07.6366162Z ok (5.520s) 2022-03-17T20:32:07.6366582Z 2022-03-17T20:32:07.6367362Z ---------------------------------------------------------------------- 2022-03-17T20:32:07.6367922Z Ran 1 test in 5.520s 2022-03-17T20:32:07.6368071Z 2022-03-17T20:32:07.6368168Z OK 2022-03-17T20:32:07.6368308Z 2022-03-17T20:32:07.6368441Z Generating XML reports... 2022-03-17T20:32:07.6412552Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203202.xml 2022-03-17T20:32:08.9401959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:08.9416872Z 2022-03-17T20:32:08.9417168Z Running tests... 2022-03-17T20:32:08.9417593Z ---------------------------------------------------------------------- 2022-03-17T20:32:10.4732431Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:10.5156819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 626 2022-03-17T20:32:10.5282102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 627 2022-03-17T20:32:11.6128651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:11.6129186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:11.6129943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:11.6130613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:11.6136655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:11.6137305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:13.9520504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:13.9521021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:13.9662093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:13.9662569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:13.9722536Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:13.9723280Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:13.9724283Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:13.9725286Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:14.3372969Z ok (5.395s) 2022-03-17T20:32:14.3373844Z 2022-03-17T20:32:14.3374260Z ---------------------------------------------------------------------- 2022-03-17T20:32:14.3374615Z Ran 1 test in 5.396s 2022-03-17T20:32:14.3374790Z 2022-03-17T20:32:14.3374952Z OK 2022-03-17T20:32:14.3375197Z 2022-03-17T20:32:14.3375386Z Generating XML reports... 2022-03-17T20:32:14.3419904Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203208.xml 2022-03-17T20:32:15.6377254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:15.6392479Z 2022-03-17T20:32:15.6392629Z Running tests... 2022-03-17T20:32:15.6393447Z ---------------------------------------------------------------------- 2022-03-17T20:32:17.1764813Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:17.2184765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 753 2022-03-17T20:32:17.2311875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 754 2022-03-17T20:32:18.2725669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:18.3067377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:18.3068218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:18.3131304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:18.3138009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:18.4082827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:20.7238294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:20.7238864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:21.1419958Z ok (5.502s) 2022-03-17T20:32:21.1420139Z 2022-03-17T20:32:21.1420967Z ---------------------------------------------------------------------- 2022-03-17T20:32:21.1421323Z Ran 1 test in 5.503s 2022-03-17T20:32:21.1421500Z 2022-03-17T20:32:21.1421594Z OK 2022-03-17T20:32:21.1421730Z 2022-03-17T20:32:21.1421837Z Generating XML reports... 2022-03-17T20:32:21.1465920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203215.xml 2022-03-17T20:32:22.4357733Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:22.4372839Z 2022-03-17T20:32:22.4373247Z Running tests... 2022-03-17T20:32:22.4373744Z ---------------------------------------------------------------------- 2022-03-17T20:32:23.9452146Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:23.9871813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 880 2022-03-17T20:32:23.9996925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 881 2022-03-17T20:32:25.0642599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:25.0744251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:25.0744823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:25.0745618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:25.0752310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:25.0753092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:28.0919315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:28.0919818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:28.1113832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:28.1121002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:28.1201625Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:28.1202443Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:28.1209699Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:28.1210485Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:28.7103410Z ok (6.273s) 2022-03-17T20:32:28.7103591Z 2022-03-17T20:32:28.7103977Z ---------------------------------------------------------------------- 2022-03-17T20:32:28.7104350Z Ran 1 test in 6.273s 2022-03-17T20:32:28.7104499Z 2022-03-17T20:32:28.7104593Z OK 2022-03-17T20:32:28.7104726Z 2022-03-17T20:32:28.7104870Z Generating XML reports... 2022-03-17T20:32:28.7149742Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203222.xml 2022-03-17T20:32:30.0081305Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:30.0096453Z 2022-03-17T20:32:30.0096822Z Running tests... 2022-03-17T20:32:30.0097222Z ---------------------------------------------------------------------- 2022-03-17T20:32:31.5314112Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:31.5737995Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1007 2022-03-17T20:32:31.5864574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1008 2022-03-17T20:32:32.6516951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:32.6519241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:32.6520274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:32.6618487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:32.6625388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:32.7534403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:35.3542751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:35.3543567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:35.3707423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:35.3708435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:35.3777757Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:35.3778551Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:35.3779782Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:35.3780857Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:35.6961594Z ok (5.686s) 2022-03-17T20:32:35.6961776Z 2022-03-17T20:32:35.6962159Z ---------------------------------------------------------------------- 2022-03-17T20:32:35.6962512Z Ran 1 test in 5.686s 2022-03-17T20:32:35.6962645Z 2022-03-17T20:32:35.6962739Z OK 2022-03-17T20:32:35.6962854Z 2022-03-17T20:32:35.6963037Z Generating XML reports... 2022-03-17T20:32:35.7007381Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203230.xml 2022-03-17T20:32:36.9923505Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:36.9938264Z 2022-03-17T20:32:36.9938678Z Running tests... 2022-03-17T20:32:36.9939210Z ---------------------------------------------------------------------- 2022-03-17T20:32:38.5391284Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:38.5814092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1134 2022-03-17T20:32:38.5943252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1135 2022-03-17T20:32:39.6677612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:39.6696936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:39.6697871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:39.6778945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:39.6785855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:39.7711313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:41.9874236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:41.9875178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:42.0008081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:42.0014310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:42.0069003Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:42.0069827Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:42.0075808Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:32:42.0076560Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:32:42.3031453Z ok (5.309s) 2022-03-17T20:32:42.3031683Z 2022-03-17T20:32:42.3032322Z ---------------------------------------------------------------------- 2022-03-17T20:32:42.3033401Z Ran 1 test in 5.309s 2022-03-17T20:32:42.3033621Z 2022-03-17T20:32:42.3033720Z OK 2022-03-17T20:32:42.3033867Z 2022-03-17T20:32:42.3033978Z Generating XML reports... 2022-03-17T20:32:42.3078511Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203236.xml 2022-03-17T20:32:43.5913657Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:43.5927827Z 2022-03-17T20:32:43.5928340Z Running tests... 2022-03-17T20:32:43.5929155Z ---------------------------------------------------------------------- 2022-03-17T20:32:45.1153022Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:45.1575409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1261 2022-03-17T20:32:45.1702175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1262 2022-03-17T20:32:46.2104704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:46.2168145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:46.2169444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:46.2208125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:46.2214690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:46.3184049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:47.9299395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:47.9300578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:49.8804903Z ok (6.287s) 2022-03-17T20:32:49.8805293Z 2022-03-17T20:32:49.8806019Z ---------------------------------------------------------------------- 2022-03-17T20:32:49.8806670Z Ran 1 test in 6.288s 2022-03-17T20:32:49.8806949Z 2022-03-17T20:32:49.8807213Z OK 2022-03-17T20:32:49.8807458Z 2022-03-17T20:32:49.8807715Z Generating XML reports... 2022-03-17T20:32:49.8851449Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203243.xml 2022-03-17T20:32:51.1738145Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:51.1753946Z 2022-03-17T20:32:51.1754142Z Running tests... 2022-03-17T20:32:51.1754600Z ---------------------------------------------------------------------- 2022-03-17T20:32:52.7053031Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:52.7474369Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1387 2022-03-17T20:32:52.7599159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1388 2022-03-17T20:32:53.8010834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:53.8080913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:53.8081910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:53.8112094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:53.8119099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:53.9094797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:54.1647914Z ok (2.989s) 2022-03-17T20:32:54.1648424Z 2022-03-17T20:32:54.1648754Z ---------------------------------------------------------------------- 2022-03-17T20:32:54.1649103Z Ran 1 test in 2.989s 2022-03-17T20:32:54.1649271Z 2022-03-17T20:32:54.1649383Z OK 2022-03-17T20:32:54.1649490Z 2022-03-17T20:32:54.1649636Z Generating XML reports... 2022-03-17T20:32:54.1694395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203251.xml 2022-03-17T20:32:55.4365671Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:32:55.4381075Z 2022-03-17T20:32:55.4381478Z Running tests... 2022-03-17T20:32:55.4381962Z ---------------------------------------------------------------------- 2022-03-17T20:32:56.9777539Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:32:57.0199386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1500 2022-03-17T20:32:57.0326487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1501 2022-03-17T20:32:58.0860086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:32:58.1262291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:32:58.1263364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:58.1265191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:32:58.1272635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:32:58.1273225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:32:59.8054013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:59.8054562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:59.8593967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:32:59.8594498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:00.2405595Z ok (4.802s) 2022-03-17T20:33:00.2405808Z 2022-03-17T20:33:00.2406194Z ---------------------------------------------------------------------- 2022-03-17T20:33:00.2406539Z Ran 1 test in 4.802s 2022-03-17T20:33:00.2406706Z 2022-03-17T20:33:00.2406782Z OK 2022-03-17T20:33:00.2406889Z 2022-03-17T20:33:00.2407030Z Generating XML reports... 2022-03-17T20:33:00.2451414Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203255.xml 2022-03-17T20:33:01.5496006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:01.5512053Z 2022-03-17T20:33:01.5512353Z Running tests... 2022-03-17T20:33:01.5512786Z ---------------------------------------------------------------------- 2022-03-17T20:33:03.0856243Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:33:03.1280274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1626 2022-03-17T20:33:03.1408093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1627 2022-03-17T20:33:04.2177974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:04.2309453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:04.2310265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:04.2381224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:04.2387924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:04.3324483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:06.9498817Z ok (5.398s) 2022-03-17T20:33:06.9499038Z 2022-03-17T20:33:06.9499443Z ---------------------------------------------------------------------- 2022-03-17T20:33:06.9499776Z Ran 1 test in 5.399s 2022-03-17T20:33:06.9499954Z 2022-03-17T20:33:06.9500038Z OK 2022-03-17T20:33:06.9500131Z 2022-03-17T20:33:06.9500517Z Generating XML reports... 2022-03-17T20:33:06.9544753Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203301.xml 2022-03-17T20:33:08.2517090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:08.2532550Z 2022-03-17T20:33:08.2532895Z Running tests... 2022-03-17T20:33:08.2533396Z ---------------------------------------------------------------------- 2022-03-17T20:33:08.2555592Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:33:08.2555989Z 2022-03-17T20:33:08.2556435Z ---------------------------------------------------------------------- 2022-03-17T20:33:08.2556769Z Ran 1 test in 0.002s 2022-03-17T20:33:08.2556942Z 2022-03-17T20:33:08.2557056Z OK (skipped=1) 2022-03-17T20:33:08.2557139Z 2022-03-17T20:33:08.2557245Z Generating XML reports... 2022-03-17T20:33:08.2601145Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203308.xml 2022-03-17T20:33:09.3970721Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:09.3986571Z 2022-03-17T20:33:09.3986803Z Running tests... 2022-03-17T20:33:09.3987238Z ---------------------------------------------------------------------- 2022-03-17T20:33:09.4005601Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:33:10.9387499Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:33:10.9810579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1784 2022-03-17T20:33:10.9937464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1785 2022-03-17T20:33:12.0711225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:12.1018950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:12.1019915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:12.1118296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:12.1124843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:12.2034087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:15.1032821Z ok (5.704s) 2022-03-17T20:33:15.1033167Z 2022-03-17T20:33:15.1033886Z ---------------------------------------------------------------------- 2022-03-17T20:33:15.1034179Z Ran 1 test in 5.705s 2022-03-17T20:33:15.1034354Z 2022-03-17T20:33:15.1034457Z OK 2022-03-17T20:33:15.1034585Z 2022-03-17T20:33:15.1034721Z Generating XML reports... 2022-03-17T20:33:15.1078067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203309.xml 2022-03-17T20:33:16.4052824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:16.4068064Z 2022-03-17T20:33:16.4068248Z Running tests... 2022-03-17T20:33:16.4068986Z ---------------------------------------------------------------------- 2022-03-17T20:33:16.4092018Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:33:17.9592901Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:33:18.0014884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1911 2022-03-17T20:33:18.0140803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1912 2022-03-17T20:33:19.0796660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:19.1135175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:19.1136303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:19.1202050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:19.1208955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:19.2150771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:21.8308773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:21.8309335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:22.2240895Z ok (5.817s) 2022-03-17T20:33:22.2241109Z 2022-03-17T20:33:22.2241490Z ---------------------------------------------------------------------- 2022-03-17T20:33:22.2241829Z Ran 1 test in 5.817s 2022-03-17T20:33:22.2242040Z 2022-03-17T20:33:22.2242113Z OK 2022-03-17T20:33:22.2242251Z 2022-03-17T20:33:22.2242389Z Generating XML reports... 2022-03-17T20:33:22.2289510Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203316.xml 2022-03-17T20:33:23.5349084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:23.5364514Z 2022-03-17T20:33:23.5364775Z Running tests... 2022-03-17T20:33:23.5365194Z ---------------------------------------------------------------------- 2022-03-17T20:33:23.5391043Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:33:25.0796809Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:33:25.1221442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2038 2022-03-17T20:33:25.1350549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2039 2022-03-17T20:33:26.2164288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:26.2249367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:26.2250212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:26.2265762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:26.2272825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:26.3263853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:28.8338620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:28.8339129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:33:29.2445503Z ok (5.708s) 2022-03-17T20:33:29.2445667Z 2022-03-17T20:33:29.2446479Z ---------------------------------------------------------------------- 2022-03-17T20:33:29.2446854Z Ran 1 test in 5.708s 2022-03-17T20:33:29.2447244Z 2022-03-17T20:33:29.2447350Z OK 2022-03-17T20:33:29.2447493Z 2022-03-17T20:33:29.2447626Z Generating XML reports... 2022-03-17T20:33:29.2494968Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203323.xml 2022-03-17T20:33:30.5464656Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:30.5481698Z 2022-03-17T20:33:30.5482141Z Running tests... 2022-03-17T20:33:30.5482622Z ---------------------------------------------------------------------- 2022-03-17T20:33:30.5502304Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:33:32.0857979Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:33:32.1282754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2165 2022-03-17T20:33:32.1410126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2166 2022-03-17T20:33:33.1976550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:33.2396920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:33.2397738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:33.2483546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:33.2490491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:33.3411363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:36.3510545Z ok (5.803s) 2022-03-17T20:33:36.3511041Z 2022-03-17T20:33:36.3511645Z ---------------------------------------------------------------------- 2022-03-17T20:33:36.3512134Z Ran 1 test in 5.803s 2022-03-17T20:33:36.3512267Z 2022-03-17T20:33:36.3512437Z OK 2022-03-17T20:33:36.3512648Z 2022-03-17T20:33:36.3512951Z Generating XML reports... 2022-03-17T20:33:36.3559816Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203330.xml 2022-03-17T20:33:37.6546088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:37.6562967Z 2022-03-17T20:33:37.6563380Z Running tests... 2022-03-17T20:33:37.6563903Z ---------------------------------------------------------------------- 2022-03-17T20:33:37.6584410Z test_all_gather (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:33:37.6584700Z 2022-03-17T20:33:37.6584944Z ---------------------------------------------------------------------- 2022-03-17T20:33:37.6585287Z Ran 1 test in 0.002s 2022-03-17T20:33:37.6585448Z 2022-03-17T20:33:37.6585552Z OK (skipped=1) 2022-03-17T20:33:37.6585715Z 2022-03-17T20:33:37.6585841Z Generating XML reports... 2022-03-17T20:33:37.6630423Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203337.xml 2022-03-17T20:33:38.8023192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:38.8038995Z 2022-03-17T20:33:38.8039449Z Running tests... 2022-03-17T20:33:38.8040054Z ---------------------------------------------------------------------- 2022-03-17T20:33:38.8060398Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:33:38.8061347Z 2022-03-17T20:33:38.8061937Z ---------------------------------------------------------------------- 2022-03-17T20:33:38.8062614Z Ran 1 test in 0.002s 2022-03-17T20:33:38.8062786Z 2022-03-17T20:33:38.8062895Z OK (skipped=1) 2022-03-17T20:33:38.8063049Z 2022-03-17T20:33:38.8063176Z Generating XML reports... 2022-03-17T20:33:38.8108070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203338.xml 2022-03-17T20:33:39.9509090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:39.9524088Z 2022-03-17T20:33:39.9524488Z Running tests... 2022-03-17T20:33:39.9525468Z ---------------------------------------------------------------------- 2022-03-17T20:33:39.9545552Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:33:39.9546026Z 2022-03-17T20:33:39.9546545Z ---------------------------------------------------------------------- 2022-03-17T20:33:39.9546983Z Ran 1 test in 0.002s 2022-03-17T20:33:39.9547134Z 2022-03-17T20:33:39.9547299Z OK (skipped=1) 2022-03-17T20:33:39.9547590Z 2022-03-17T20:33:39.9547840Z Generating XML reports... 2022-03-17T20:33:39.9592197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203339.xml 2022-03-17T20:33:41.0986079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:41.1001820Z 2022-03-17T20:33:41.1001960Z Running tests... 2022-03-17T20:33:41.1002789Z ---------------------------------------------------------------------- 2022-03-17T20:33:41.1022527Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:33:41.1023125Z 2022-03-17T20:33:41.1023434Z ---------------------------------------------------------------------- 2022-03-17T20:33:41.1023805Z Ran 1 test in 0.002s 2022-03-17T20:33:41.1023982Z 2022-03-17T20:33:41.1024075Z OK (skipped=1) 2022-03-17T20:33:41.1024231Z 2022-03-17T20:33:41.1024352Z Generating XML reports... 2022-03-17T20:33:41.1069466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203341.xml 2022-03-17T20:33:42.2516192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:42.2531840Z 2022-03-17T20:33:42.2532100Z Running tests... 2022-03-17T20:33:42.2532591Z ---------------------------------------------------------------------- 2022-03-17T20:33:42.2553510Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:33:42.2553785Z 2022-03-17T20:33:42.2554320Z ---------------------------------------------------------------------- 2022-03-17T20:33:42.2554686Z Ran 1 test in 0.002s 2022-03-17T20:33:42.2554834Z 2022-03-17T20:33:42.2554941Z OK (skipped=1) 2022-03-17T20:33:42.2555096Z 2022-03-17T20:33:42.2555222Z Generating XML reports... 2022-03-17T20:33:42.2599804Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203342.xml 2022-03-17T20:33:43.4056151Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:43.4072269Z 2022-03-17T20:33:43.4072668Z Running tests... 2022-03-17T20:33:43.4073110Z ---------------------------------------------------------------------- 2022-03-17T20:33:43.4104945Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:33:43.4105364Z 2022-03-17T20:33:43.4105880Z ---------------------------------------------------------------------- 2022-03-17T20:33:43.4106429Z Ran 1 test in 0.003s 2022-03-17T20:33:43.4106606Z 2022-03-17T20:33:43.4106701Z OK (skipped=1) 2022-03-17T20:33:43.4106859Z 2022-03-17T20:33:43.4106985Z Generating XML reports... 2022-03-17T20:33:43.4151334Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203343.xml 2022-03-17T20:33:44.5548018Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:44.5563361Z 2022-03-17T20:33:44.5563779Z Running tests... 2022-03-17T20:33:44.5564247Z ---------------------------------------------------------------------- 2022-03-17T20:33:44.5584172Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:33:44.5584686Z 2022-03-17T20:33:44.5585237Z ---------------------------------------------------------------------- 2022-03-17T20:33:44.5585618Z Ran 1 test in 0.002s 2022-03-17T20:33:44.5585709Z 2022-03-17T20:33:44.5585828Z OK (skipped=1) 2022-03-17T20:33:44.5585998Z 2022-03-17T20:33:44.5586104Z Generating XML reports... 2022-03-17T20:33:44.5630361Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203344.xml 2022-03-17T20:33:45.7085731Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:45.7101105Z 2022-03-17T20:33:45.7101253Z Running tests... 2022-03-17T20:33:45.7102027Z ---------------------------------------------------------------------- 2022-03-17T20:33:47.2706671Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:33:47.3129435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2537 2022-03-17T20:33:47.3257385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2538 2022-03-17T20:33:48.3660856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:48.3828049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:48.3828856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:48.3864683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:48.3871791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:48.4843661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:33:52.4368815Z ok (6.726s) 2022-03-17T20:33:52.4369032Z 2022-03-17T20:33:52.4369433Z ---------------------------------------------------------------------- 2022-03-17T20:33:52.4369712Z Ran 1 test in 6.727s 2022-03-17T20:33:52.4369856Z 2022-03-17T20:33:52.4370236Z OK 2022-03-17T20:33:52.4370428Z 2022-03-17T20:33:52.4370551Z Generating XML reports... 2022-03-17T20:33:52.4414909Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203345.xml 2022-03-17T20:33:53.7386185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:33:53.7401847Z 2022-03-17T20:33:53.7401983Z Running tests... 2022-03-17T20:33:53.7402866Z ---------------------------------------------------------------------- 2022-03-17T20:33:55.2711936Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:33:55.3136417Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2665 2022-03-17T20:33:55.3262475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2666 2022-03-17T20:33:56.4075309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:33:56.4148786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:33:56.4150035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:56.4176353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:33:56.4183094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:33:56.5164846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:34:00.5378563Z ok (6.797s) 2022-03-17T20:34:00.5378827Z 2022-03-17T20:34:00.5379541Z ---------------------------------------------------------------------- 2022-03-17T20:34:00.5379874Z Ran 1 test in 6.798s 2022-03-17T20:34:00.5380021Z 2022-03-17T20:34:00.5380114Z OK 2022-03-17T20:34:00.5380524Z 2022-03-17T20:34:00.5381067Z Generating XML reports... 2022-03-17T20:34:00.5425204Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203353.xml 2022-03-17T20:34:01.8350294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:01.8365373Z 2022-03-17T20:34:01.8365887Z Running tests... 2022-03-17T20:34:01.8366361Z ---------------------------------------------------------------------- 2022-03-17T20:34:01.8385998Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:01.8386308Z 2022-03-17T20:34:01.8386568Z ---------------------------------------------------------------------- 2022-03-17T20:34:01.8386881Z Ran 1 test in 0.002s 2022-03-17T20:34:01.8387050Z 2022-03-17T20:34:01.8387184Z OK (skipped=1) 2022-03-17T20:34:01.8387343Z 2022-03-17T20:34:01.8387451Z Generating XML reports... 2022-03-17T20:34:01.8430798Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203401.xml 2022-03-17T20:34:02.9888022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:02.9903594Z 2022-03-17T20:34:02.9903720Z Running tests... 2022-03-17T20:34:02.9904645Z ---------------------------------------------------------------------- 2022-03-17T20:34:02.9924918Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:02.9925187Z 2022-03-17T20:34:02.9925488Z ---------------------------------------------------------------------- 2022-03-17T20:34:02.9925803Z Ran 1 test in 0.002s 2022-03-17T20:34:02.9925970Z 2022-03-17T20:34:02.9926080Z OK (skipped=1) 2022-03-17T20:34:02.9926244Z 2022-03-17T20:34:02.9926373Z Generating XML reports... 2022-03-17T20:34:02.9970368Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203402.xml 2022-03-17T20:34:04.1261348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:04.1276998Z 2022-03-17T20:34:04.1277170Z Running tests... 2022-03-17T20:34:04.1277868Z ---------------------------------------------------------------------- 2022-03-17T20:34:05.6393751Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:34:05.6812504Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2863 2022-03-17T20:34:05.6937763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2864 2022-03-17T20:34:06.7634934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:34:06.7800996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:34:06.7801785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:06.7837472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:06.7844408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:34:06.8816604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:34:09.0017707Z ok (4.874s) 2022-03-17T20:34:09.0018141Z 2022-03-17T20:34:09.0018629Z ---------------------------------------------------------------------- 2022-03-17T20:34:09.0018861Z Ran 1 test in 4.874s 2022-03-17T20:34:09.0019029Z 2022-03-17T20:34:09.0019124Z OK 2022-03-17T20:34:09.0019259Z 2022-03-17T20:34:09.0019378Z Generating XML reports... 2022-03-17T20:34:09.0064444Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203404.xml 2022-03-17T20:34:10.3042122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:10.3058283Z 2022-03-17T20:34:10.3058697Z Running tests... 2022-03-17T20:34:10.3059163Z ---------------------------------------------------------------------- 2022-03-17T20:34:11.8389324Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:34:11.8811505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2989 2022-03-17T20:34:11.8936334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2990 2022-03-17T20:34:12.9627864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:34:12.9986412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:34:12.9987450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:13.0032886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:13.0039747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:34:13.1001286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:34:15.3026810Z ok (4.996s) 2022-03-17T20:34:15.3027048Z 2022-03-17T20:34:15.3027398Z ---------------------------------------------------------------------- 2022-03-17T20:34:15.3027716Z Ran 1 test in 4.997s 2022-03-17T20:34:15.3027886Z 2022-03-17T20:34:15.3027987Z OK 2022-03-17T20:34:15.3028123Z 2022-03-17T20:34:15.3028241Z Generating XML reports... 2022-03-17T20:34:15.3081867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203410.xml 2022-03-17T20:34:16.6077690Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:16.6092667Z 2022-03-17T20:34:16.6093260Z Running tests... 2022-03-17T20:34:16.6093755Z ---------------------------------------------------------------------- 2022-03-17T20:34:18.1494985Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:34:18.1917932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3115 2022-03-17T20:34:18.2049435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3116 2022-03-17T20:34:19.2402026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:34:19.2417883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:34:19.2418967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:19.2504530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:19.2512148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:34:19.3434926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:34:21.9138328Z ok (5.304s) 2022-03-17T20:34:21.9138562Z 2022-03-17T20:34:21.9138978Z ---------------------------------------------------------------------- 2022-03-17T20:34:21.9139325Z Ran 1 test in 5.305s 2022-03-17T20:34:21.9139473Z 2022-03-17T20:34:21.9139570Z OK 2022-03-17T20:34:21.9139706Z 2022-03-17T20:34:21.9139838Z Generating XML reports... 2022-03-17T20:34:21.9186153Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203416.xml 2022-03-17T20:34:23.2228200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:23.2242992Z 2022-03-17T20:34:23.2243141Z Running tests... 2022-03-17T20:34:23.2244047Z ---------------------------------------------------------------------- 2022-03-17T20:34:24.7583107Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:34:24.8007095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3238 2022-03-17T20:34:24.8136318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3239 2022-03-17T20:34:25.8716908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:34:25.8850793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:34:25.8851649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:25.8919846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:34:25.8926711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:34:25.9864593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:34:26.0048162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:34:26.0048655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:34:26.0049344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:34:26.0050039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:34:29.4445319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:34:29.4445895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:34:29.4446685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:34:29.4447378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:34:29.4898033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:34:29.4898593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:34:29.4899374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:34:29.4900068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:34:29.9251532Z ok (6.700s) 2022-03-17T20:34:29.9251754Z 2022-03-17T20:34:29.9252132Z ---------------------------------------------------------------------- 2022-03-17T20:34:29.9252474Z Ran 1 test in 6.701s 2022-03-17T20:34:29.9252645Z 2022-03-17T20:34:29.9252719Z OK 2022-03-17T20:34:29.9253099Z 2022-03-17T20:34:29.9253232Z Generating XML reports... 2022-03-17T20:34:29.9298106Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203423.xml 2022-03-17T20:34:31.2211478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:31.2226816Z 2022-03-17T20:34:31.2226958Z Running tests... 2022-03-17T20:34:31.2227431Z ---------------------------------------------------------------------- 2022-03-17T20:34:31.2247178Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:31.2247437Z 2022-03-17T20:34:31.2248065Z ---------------------------------------------------------------------- 2022-03-17T20:34:31.2248412Z Ran 1 test in 0.002s 2022-03-17T20:34:31.2248560Z 2022-03-17T20:34:31.2248677Z OK (skipped=1) 2022-03-17T20:34:31.2248833Z 2022-03-17T20:34:31.2248961Z Generating XML reports... 2022-03-17T20:34:31.2293048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203431.xml 2022-03-17T20:34:32.3678792Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:32.3693665Z 2022-03-17T20:34:32.3693960Z Running tests... 2022-03-17T20:34:32.3694653Z ---------------------------------------------------------------------- 2022-03-17T20:34:32.3715210Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:32.3715500Z 2022-03-17T20:34:32.3715803Z ---------------------------------------------------------------------- 2022-03-17T20:34:32.3716346Z Ran 1 test in 0.002s 2022-03-17T20:34:32.3716529Z 2022-03-17T20:34:32.3716640Z OK (skipped=1) 2022-03-17T20:34:32.3716795Z 2022-03-17T20:34:32.3716900Z Generating XML reports... 2022-03-17T20:34:32.3760258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203432.xml 2022-03-17T20:34:33.5203703Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:33.5218743Z 2022-03-17T20:34:33.5219212Z Running tests... 2022-03-17T20:34:33.5219922Z ---------------------------------------------------------------------- 2022-03-17T20:34:33.5240526Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:33.5240970Z 2022-03-17T20:34:33.5241261Z ---------------------------------------------------------------------- 2022-03-17T20:34:33.5241570Z Ran 1 test in 0.002s 2022-03-17T20:34:33.5241783Z 2022-03-17T20:34:33.5241980Z OK (skipped=1) 2022-03-17T20:34:33.5242273Z 2022-03-17T20:34:33.5242426Z Generating XML reports... 2022-03-17T20:34:33.5285963Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203433.xml 2022-03-17T20:34:34.6653579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:34.6668517Z 2022-03-17T20:34:34.6668663Z Running tests... 2022-03-17T20:34:34.6669439Z ---------------------------------------------------------------------- 2022-03-17T20:34:34.6689502Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:34.6689848Z 2022-03-17T20:34:34.6690117Z ---------------------------------------------------------------------- 2022-03-17T20:34:34.6690430Z Ran 1 test in 0.002s 2022-03-17T20:34:34.6690595Z 2022-03-17T20:34:34.6690711Z OK (skipped=1) 2022-03-17T20:34:34.6690844Z 2022-03-17T20:34:34.6690972Z Generating XML reports... 2022-03-17T20:34:34.6734215Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203434.xml 2022-03-17T20:34:35.8141048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:35.8155488Z 2022-03-17T20:34:35.8155842Z Running tests... 2022-03-17T20:34:35.8157073Z ---------------------------------------------------------------------- 2022-03-17T20:34:35.8176072Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:35.8176698Z 2022-03-17T20:34:35.8177279Z ---------------------------------------------------------------------- 2022-03-17T20:34:35.8177761Z Ran 1 test in 0.002s 2022-03-17T20:34:35.8177915Z 2022-03-17T20:34:35.8178013Z OK (skipped=1) 2022-03-17T20:34:35.8178170Z 2022-03-17T20:34:35.8178297Z Generating XML reports... 2022-03-17T20:34:35.8221666Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203435.xml 2022-03-17T20:34:36.9509084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:36.9522932Z 2022-03-17T20:34:36.9523309Z Running tests... 2022-03-17T20:34:36.9524233Z ---------------------------------------------------------------------- 2022-03-17T20:34:36.9543938Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:36.9544504Z 2022-03-17T20:34:36.9545094Z ---------------------------------------------------------------------- 2022-03-17T20:34:36.9545616Z Ran 1 test in 0.002s 2022-03-17T20:34:36.9545786Z 2022-03-17T20:34:36.9545895Z OK (skipped=1) 2022-03-17T20:34:36.9546050Z 2022-03-17T20:34:36.9546155Z Generating XML reports... 2022-03-17T20:34:36.9588586Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203436.xml 2022-03-17T20:34:38.1017364Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:38.1032794Z 2022-03-17T20:34:38.1033043Z Running tests... 2022-03-17T20:34:38.1033500Z ---------------------------------------------------------------------- 2022-03-17T20:34:38.1053790Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:38.1054072Z 2022-03-17T20:34:38.1054352Z ---------------------------------------------------------------------- 2022-03-17T20:34:38.1054677Z Ran 1 test in 0.002s 2022-03-17T20:34:38.1054845Z 2022-03-17T20:34:38.1054948Z OK (skipped=1) 2022-03-17T20:34:38.1055088Z 2022-03-17T20:34:38.1055207Z Generating XML reports... 2022-03-17T20:34:38.1098803Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203438.xml 2022-03-17T20:34:39.2458928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:39.2474902Z 2022-03-17T20:34:39.2475052Z Running tests... 2022-03-17T20:34:39.2475543Z ---------------------------------------------------------------------- 2022-03-17T20:34:39.2495711Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:39.2496031Z 2022-03-17T20:34:39.2496615Z ---------------------------------------------------------------------- 2022-03-17T20:34:39.2496982Z Ran 1 test in 0.002s 2022-03-17T20:34:39.2497162Z 2022-03-17T20:34:39.2497276Z OK (skipped=1) 2022-03-17T20:34:39.2497444Z 2022-03-17T20:34:39.2497554Z Generating XML reports... 2022-03-17T20:34:39.2540729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203439.xml 2022-03-17T20:34:40.3895068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:40.3910602Z 2022-03-17T20:34:40.3911051Z Running tests... 2022-03-17T20:34:40.3911561Z ---------------------------------------------------------------------- 2022-03-17T20:34:40.3931543Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:40.3931889Z 2022-03-17T20:34:40.3932377Z ---------------------------------------------------------------------- 2022-03-17T20:34:40.3932827Z Ran 1 test in 0.002s 2022-03-17T20:34:40.3933001Z 2022-03-17T20:34:40.3933373Z OK (skipped=1) 2022-03-17T20:34:40.3933515Z 2022-03-17T20:34:40.3933640Z Generating XML reports... 2022-03-17T20:34:40.3976781Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203440.xml 2022-03-17T20:34:41.5381911Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:41.5397261Z 2022-03-17T20:34:41.5397369Z Running tests... 2022-03-17T20:34:41.5398033Z ---------------------------------------------------------------------- 2022-03-17T20:34:41.5418726Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:41.5419087Z 2022-03-17T20:34:41.5419765Z ---------------------------------------------------------------------- 2022-03-17T20:34:41.5420099Z Ran 1 test in 0.002s 2022-03-17T20:34:41.5420609Z 2022-03-17T20:34:41.5420708Z OK (skipped=1) 2022-03-17T20:34:41.5420872Z 2022-03-17T20:34:41.5420996Z Generating XML reports... 2022-03-17T20:34:41.5463927Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203441.xml 2022-03-17T20:34:42.6818501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:42.6833510Z 2022-03-17T20:34:42.6833787Z Running tests... 2022-03-17T20:34:42.6834200Z ---------------------------------------------------------------------- 2022-03-17T20:34:42.6854493Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:42.6854732Z 2022-03-17T20:34:42.6855056Z ---------------------------------------------------------------------- 2022-03-17T20:34:42.6855589Z Ran 1 test in 0.002s 2022-03-17T20:34:42.6855766Z 2022-03-17T20:34:42.6855878Z OK (skipped=1) 2022-03-17T20:34:42.6856035Z 2022-03-17T20:34:42.6856140Z Generating XML reports... 2022-03-17T20:34:42.6899939Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203442.xml 2022-03-17T20:34:43.8325250Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:43.8340212Z 2022-03-17T20:34:43.8340887Z Running tests... 2022-03-17T20:34:43.8341348Z ---------------------------------------------------------------------- 2022-03-17T20:34:43.8361853Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:43.8362180Z 2022-03-17T20:34:43.8362519Z ---------------------------------------------------------------------- 2022-03-17T20:34:43.8362834Z Ran 1 test in 0.002s 2022-03-17T20:34:43.8363002Z 2022-03-17T20:34:43.8363115Z OK (skipped=1) 2022-03-17T20:34:43.8363290Z 2022-03-17T20:34:43.8363407Z Generating XML reports... 2022-03-17T20:34:43.8407243Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203443.xml 2022-03-17T20:34:44.9811128Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:44.9826104Z 2022-03-17T20:34:44.9826242Z Running tests... 2022-03-17T20:34:44.9826999Z ---------------------------------------------------------------------- 2022-03-17T20:34:44.9846758Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:44.9847046Z 2022-03-17T20:34:44.9847296Z ---------------------------------------------------------------------- 2022-03-17T20:34:44.9847643Z Ran 1 test in 0.002s 2022-03-17T20:34:44.9847812Z 2022-03-17T20:34:44.9847915Z OK (skipped=1) 2022-03-17T20:34:44.9848061Z 2022-03-17T20:34:44.9848173Z Generating XML reports... 2022-03-17T20:34:44.9892524Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203444.xml 2022-03-17T20:34:46.1311322Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:46.1326579Z 2022-03-17T20:34:46.1326709Z Running tests... 2022-03-17T20:34:46.1327458Z ---------------------------------------------------------------------- 2022-03-17T20:34:46.1350792Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:46.1351071Z 2022-03-17T20:34:46.1351361Z ---------------------------------------------------------------------- 2022-03-17T20:34:46.1351698Z Ran 1 test in 0.002s 2022-03-17T20:34:46.1351868Z 2022-03-17T20:34:46.1351959Z OK (skipped=1) 2022-03-17T20:34:46.1352114Z 2022-03-17T20:34:46.1352241Z Generating XML reports... 2022-03-17T20:34:46.1395676Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203446.xml 2022-03-17T20:34:47.2766965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:47.2782319Z 2022-03-17T20:34:47.2782620Z Running tests... 2022-03-17T20:34:47.2783048Z ---------------------------------------------------------------------- 2022-03-17T20:34:47.2803559Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:47.2803843Z 2022-03-17T20:34:47.2804132Z ---------------------------------------------------------------------- 2022-03-17T20:34:47.2804454Z Ran 1 test in 0.002s 2022-03-17T20:34:47.2804620Z 2022-03-17T20:34:47.2804719Z OK (skipped=1) 2022-03-17T20:34:47.2804856Z 2022-03-17T20:34:47.2804982Z Generating XML reports... 2022-03-17T20:34:47.2848965Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203447.xml 2022-03-17T20:34:48.4201784Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:48.4217603Z 2022-03-17T20:34:48.4217802Z Running tests... 2022-03-17T20:34:48.4218229Z ---------------------------------------------------------------------- 2022-03-17T20:34:48.4238343Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:48.4238627Z 2022-03-17T20:34:48.4238956Z ---------------------------------------------------------------------- 2022-03-17T20:34:48.4239238Z Ran 1 test in 0.002s 2022-03-17T20:34:48.4239386Z 2022-03-17T20:34:48.4239494Z OK (skipped=1) 2022-03-17T20:34:48.4239631Z 2022-03-17T20:34:48.4239754Z Generating XML reports... 2022-03-17T20:34:48.4284009Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203448.xml 2022-03-17T20:34:49.5675991Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:49.5691890Z 2022-03-17T20:34:49.5692254Z Running tests... 2022-03-17T20:34:49.5692743Z ---------------------------------------------------------------------- 2022-03-17T20:34:49.5714782Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:49.5715087Z 2022-03-17T20:34:49.5715344Z ---------------------------------------------------------------------- 2022-03-17T20:34:49.5715684Z Ran 1 test in 0.002s 2022-03-17T20:34:49.5715849Z 2022-03-17T20:34:49.5715958Z OK (skipped=1) 2022-03-17T20:34:49.5716098Z 2022-03-17T20:34:49.5716223Z Generating XML reports... 2022-03-17T20:34:49.5760701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203449.xml 2022-03-17T20:34:50.7235039Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:50.7250659Z 2022-03-17T20:34:50.7250859Z Running tests... 2022-03-17T20:34:50.7251337Z ---------------------------------------------------------------------- 2022-03-17T20:34:50.7273312Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:50.7273606Z 2022-03-17T20:34:50.7274132Z ---------------------------------------------------------------------- 2022-03-17T20:34:50.7274454Z Ran 1 test in 0.002s 2022-03-17T20:34:50.7274618Z 2022-03-17T20:34:50.7274998Z OK (skipped=1) 2022-03-17T20:34:50.7275159Z 2022-03-17T20:34:50.7275282Z Generating XML reports... 2022-03-17T20:34:50.7318549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203450.xml 2022-03-17T20:34:51.8738910Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:51.8754660Z 2022-03-17T20:34:51.8754872Z Running tests... 2022-03-17T20:34:51.8755334Z ---------------------------------------------------------------------- 2022-03-17T20:34:51.8776029Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:51.8776333Z 2022-03-17T20:34:51.8776984Z ---------------------------------------------------------------------- 2022-03-17T20:34:51.8777328Z Ran 1 test in 0.002s 2022-03-17T20:34:51.8777490Z 2022-03-17T20:34:51.8777583Z OK (skipped=1) 2022-03-17T20:34:51.8777738Z 2022-03-17T20:34:51.8777861Z Generating XML reports... 2022-03-17T20:34:51.8822071Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203451.xml 2022-03-17T20:34:53.0217678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:53.0233563Z 2022-03-17T20:34:53.0233989Z Running tests... 2022-03-17T20:34:53.0234455Z ---------------------------------------------------------------------- 2022-03-17T20:34:53.0254663Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:53.0254936Z 2022-03-17T20:34:53.0255278Z ---------------------------------------------------------------------- 2022-03-17T20:34:53.0255543Z Ran 1 test in 0.002s 2022-03-17T20:34:53.0255729Z 2022-03-17T20:34:53.0255833Z OK (skipped=1) 2022-03-17T20:34:53.0255988Z 2022-03-17T20:34:53.0256109Z Generating XML reports... 2022-03-17T20:34:53.0300312Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203453.xml 2022-03-17T20:34:54.1791885Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:54.1807305Z 2022-03-17T20:34:54.1807684Z Running tests... 2022-03-17T20:34:54.1808165Z ---------------------------------------------------------------------- 2022-03-17T20:34:54.1829984Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:54.1830275Z 2022-03-17T20:34:54.1830576Z ---------------------------------------------------------------------- 2022-03-17T20:34:54.1830883Z Ran 1 test in 0.002s 2022-03-17T20:34:54.1831047Z 2022-03-17T20:34:54.1831149Z OK (skipped=1) 2022-03-17T20:34:54.1831301Z 2022-03-17T20:34:54.1831445Z Generating XML reports... 2022-03-17T20:34:54.1875944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203454.xml 2022-03-17T20:34:55.3249149Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:55.3264345Z 2022-03-17T20:34:55.3264768Z Running tests... 2022-03-17T20:34:55.3265253Z ---------------------------------------------------------------------- 2022-03-17T20:34:55.3286143Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:55.3286428Z 2022-03-17T20:34:55.3286721Z ---------------------------------------------------------------------- 2022-03-17T20:34:55.3287095Z Ran 1 test in 0.002s 2022-03-17T20:34:55.3287243Z 2022-03-17T20:34:55.3287297Z OK (skipped=1) 2022-03-17T20:34:55.3287444Z 2022-03-17T20:34:55.3288866Z Generating XML reports... 2022-03-17T20:34:55.3331095Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203455.xml 2022-03-17T20:34:56.4707331Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:56.4722021Z 2022-03-17T20:34:56.4722432Z Running tests... 2022-03-17T20:34:56.4723210Z ---------------------------------------------------------------------- 2022-03-17T20:34:56.4743210Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:56.4743522Z 2022-03-17T20:34:56.4743766Z ---------------------------------------------------------------------- 2022-03-17T20:34:56.4744099Z Ran 1 test in 0.002s 2022-03-17T20:34:56.4744246Z 2022-03-17T20:34:56.4744362Z OK (skipped=1) 2022-03-17T20:34:56.4744509Z 2022-03-17T20:34:56.4744674Z Generating XML reports... 2022-03-17T20:34:56.4788941Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203456.xml 2022-03-17T20:34:57.5709849Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:57.5724923Z 2022-03-17T20:34:57.5725221Z Running tests... 2022-03-17T20:34:57.5725736Z ---------------------------------------------------------------------- 2022-03-17T20:34:57.5746332Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:57.5746630Z 2022-03-17T20:34:57.5747185Z ---------------------------------------------------------------------- 2022-03-17T20:34:57.5747841Z Ran 1 test in 0.002s 2022-03-17T20:34:57.5748064Z 2022-03-17T20:34:57.5748246Z OK (skipped=1) 2022-03-17T20:34:57.5748353Z 2022-03-17T20:34:57.5748463Z Generating XML reports... 2022-03-17T20:34:57.5792644Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203457.xml 2022-03-17T20:34:58.7201186Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:58.7216253Z 2022-03-17T20:34:58.7216685Z Running tests... 2022-03-17T20:34:58.7217167Z ---------------------------------------------------------------------- 2022-03-17T20:34:58.7240545Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:58.7240746Z 2022-03-17T20:34:58.7241045Z ---------------------------------------------------------------------- 2022-03-17T20:34:58.7241379Z Ran 1 test in 0.002s 2022-03-17T20:34:58.7241519Z 2022-03-17T20:34:58.7241634Z OK (skipped=1) 2022-03-17T20:34:58.7241850Z 2022-03-17T20:34:58.7241981Z Generating XML reports... 2022-03-17T20:34:58.7286018Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203458.xml 2022-03-17T20:34:59.8702308Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:34:59.8717798Z 2022-03-17T20:34:59.8718051Z Running tests... 2022-03-17T20:34:59.8718538Z ---------------------------------------------------------------------- 2022-03-17T20:34:59.8743195Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:34:59.8743426Z 2022-03-17T20:34:59.8743704Z ---------------------------------------------------------------------- 2022-03-17T20:34:59.8744075Z Ran 1 test in 0.003s 2022-03-17T20:34:59.8744243Z 2022-03-17T20:34:59.8744711Z OK (skipped=1) 2022-03-17T20:34:59.8744872Z 2022-03-17T20:34:59.8745361Z Generating XML reports... 2022-03-17T20:34:59.8789242Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203459.xml 2022-03-17T20:35:01.0298813Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:01.0314883Z 2022-03-17T20:35:01.0315151Z Running tests... 2022-03-17T20:35:01.0315573Z ---------------------------------------------------------------------- 2022-03-17T20:35:01.0337336Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:01.0337615Z 2022-03-17T20:35:01.0337843Z ---------------------------------------------------------------------- 2022-03-17T20:35:01.0338154Z Ran 1 test in 0.002s 2022-03-17T20:35:01.0338320Z 2022-03-17T20:35:01.0338435Z OK (skipped=1) 2022-03-17T20:35:01.0338650Z 2022-03-17T20:35:01.0339004Z Generating XML reports... 2022-03-17T20:35:01.0384263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203501.xml 2022-03-17T20:35:02.1876811Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:02.1892513Z 2022-03-17T20:35:02.1892897Z Running tests... 2022-03-17T20:35:02.1893372Z ---------------------------------------------------------------------- 2022-03-17T20:35:03.7360566Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:35:03.7783872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4346 2022-03-17T20:35:03.7910692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4347 2022-03-17T20:35:04.8670948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:04.8803563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:04.8804336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:04.8873412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:04.8880049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:04.9819190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:07.4999920Z ok (5.310s) 2022-03-17T20:35:07.5000066Z 2022-03-17T20:35:07.5000480Z ---------------------------------------------------------------------- 2022-03-17T20:35:07.5000821Z Ran 1 test in 5.311s 2022-03-17T20:35:07.5001013Z 2022-03-17T20:35:07.5001189Z OK 2022-03-17T20:35:07.5001303Z 2022-03-17T20:35:07.5001366Z Generating XML reports... 2022-03-17T20:35:07.5045190Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203502.xml 2022-03-17T20:35:08.7992046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:08.8006652Z 2022-03-17T20:35:08.8006965Z Running tests... 2022-03-17T20:35:08.8007913Z ---------------------------------------------------------------------- 2022-03-17T20:35:08.8030118Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:08.8030661Z 2022-03-17T20:35:08.8031266Z ---------------------------------------------------------------------- 2022-03-17T20:35:08.8031758Z Ran 1 test in 0.002s 2022-03-17T20:35:08.8031906Z 2022-03-17T20:35:08.8032033Z OK (skipped=1) 2022-03-17T20:35:08.8032189Z 2022-03-17T20:35:08.8032312Z Generating XML reports... 2022-03-17T20:35:08.8074583Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203508.xml 2022-03-17T20:35:09.9466464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:09.9481251Z 2022-03-17T20:35:09.9481669Z Running tests... 2022-03-17T20:35:09.9482635Z ---------------------------------------------------------------------- 2022-03-17T20:35:09.9504458Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:09.9505086Z 2022-03-17T20:35:09.9505611Z ---------------------------------------------------------------------- 2022-03-17T20:35:09.9506119Z Ran 1 test in 0.002s 2022-03-17T20:35:09.9506284Z 2022-03-17T20:35:09.9506397Z OK (skipped=1) 2022-03-17T20:35:09.9506535Z 2022-03-17T20:35:09.9506661Z Generating XML reports... 2022-03-17T20:35:09.9550141Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203509.xml 2022-03-17T20:35:11.1028577Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:11.1044315Z 2022-03-17T20:35:11.1044892Z Running tests... 2022-03-17T20:35:11.1045400Z ---------------------------------------------------------------------- 2022-03-17T20:35:11.1067870Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:11.1068170Z 2022-03-17T20:35:11.1068514Z ---------------------------------------------------------------------- 2022-03-17T20:35:11.1068855Z Ran 1 test in 0.002s 2022-03-17T20:35:11.1069020Z 2022-03-17T20:35:11.1069114Z OK (skipped=1) 2022-03-17T20:35:11.1069265Z 2022-03-17T20:35:11.1069333Z Generating XML reports... 2022-03-17T20:35:11.1113722Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203511.xml 2022-03-17T20:35:12.2646137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:12.2660934Z 2022-03-17T20:35:12.2661471Z Running tests... 2022-03-17T20:35:12.2661961Z ---------------------------------------------------------------------- 2022-03-17T20:35:13.8085956Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:35:13.8508216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4574 2022-03-17T20:35:13.8634781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4575 2022-03-17T20:35:14.9334398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:14.9553218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:14.9554184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:14.9639058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:14.9645773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:15.0568910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:17.1718852Z ok (4.905s) 2022-03-17T20:35:17.1719192Z 2022-03-17T20:35:17.1719742Z ---------------------------------------------------------------------- 2022-03-17T20:35:17.1720038Z Ran 1 test in 4.906s 2022-03-17T20:35:17.1720203Z 2022-03-17T20:35:17.1720294Z OK 2022-03-17T20:35:17.1720412Z 2022-03-17T20:35:17.1720546Z Generating XML reports... 2022-03-17T20:35:17.1764178Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203512.xml 2022-03-17T20:35:18.4206610Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:18.4221818Z 2022-03-17T20:35:18.4222040Z Running tests... 2022-03-17T20:35:18.4222460Z ---------------------------------------------------------------------- 2022-03-17T20:35:19.9743855Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:35:20.0164706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4700 2022-03-17T20:35:20.0290694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4701 2022-03-17T20:35:21.1222890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:21.1381479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:21.1382236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:21.1425496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:21.1431962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:21.2396011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:23.3372266Z ok (4.915s) 2022-03-17T20:35:23.3372470Z 2022-03-17T20:35:23.3372863Z ---------------------------------------------------------------------- 2022-03-17T20:35:23.3373197Z Ran 1 test in 4.915s 2022-03-17T20:35:23.3373361Z 2022-03-17T20:35:23.3373433Z OK 2022-03-17T20:35:23.3373567Z 2022-03-17T20:35:23.3373700Z Generating XML reports... 2022-03-17T20:35:23.3417157Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203518.xml 2022-03-17T20:35:24.6363268Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:24.6378197Z 2022-03-17T20:35:24.6378655Z Running tests... 2022-03-17T20:35:24.6379136Z ---------------------------------------------------------------------- 2022-03-17T20:35:26.1691479Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:35:26.2112296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4826 2022-03-17T20:35:26.2238928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4827 2022-03-17T20:35:27.2622294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:27.2738224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:27.2738984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:27.2825904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:27.2833407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:27.3753849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:29.4319250Z ok (4.794s) 2022-03-17T20:35:29.4319550Z 2022-03-17T20:35:29.4319959Z ---------------------------------------------------------------------- 2022-03-17T20:35:29.4320291Z Ran 1 test in 4.794s 2022-03-17T20:35:29.4320399Z 2022-03-17T20:35:29.4320493Z OK 2022-03-17T20:35:29.4320625Z 2022-03-17T20:35:29.4320758Z Generating XML reports... 2022-03-17T20:35:29.4364470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203524.xml 2022-03-17T20:35:30.7351538Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:30.7367132Z 2022-03-17T20:35:30.7367397Z Running tests... 2022-03-17T20:35:30.7367860Z ---------------------------------------------------------------------- 2022-03-17T20:35:30.7387271Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:30.7387528Z 2022-03-17T20:35:30.7387815Z ---------------------------------------------------------------------- 2022-03-17T20:35:30.7388136Z Ran 1 test in 0.002s 2022-03-17T20:35:30.7388297Z 2022-03-17T20:35:30.7388408Z OK (skipped=1) 2022-03-17T20:35:30.7388561Z 2022-03-17T20:35:30.7388668Z Generating XML reports... 2022-03-17T20:35:30.7437671Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203530.xml 2022-03-17T20:35:31.8891066Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:31.8906331Z 2022-03-17T20:35:31.8906593Z Running tests... 2022-03-17T20:35:31.8907042Z ---------------------------------------------------------------------- 2022-03-17T20:35:31.8926449Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:31.8926717Z 2022-03-17T20:35:31.8926970Z ---------------------------------------------------------------------- 2022-03-17T20:35:31.8927300Z Ran 1 test in 0.002s 2022-03-17T20:35:31.8927747Z 2022-03-17T20:35:31.8927856Z OK (skipped=1) 2022-03-17T20:35:31.8927995Z 2022-03-17T20:35:31.8928119Z Generating XML reports... 2022-03-17T20:35:31.8972112Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203531.xml 2022-03-17T20:35:33.0384822Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:33.0399627Z 2022-03-17T20:35:33.0400094Z Running tests... 2022-03-17T20:35:33.0400581Z ---------------------------------------------------------------------- 2022-03-17T20:35:34.5624548Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:35:34.6044746Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5022 2022-03-17T20:35:34.6169347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5023 2022-03-17T20:35:35.6585454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:35.6586413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:35.6587821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:35.6589164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:35.6594541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:35.6595469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:38.3261472Z ok (5.286s) 2022-03-17T20:35:38.3261663Z 2022-03-17T20:35:38.3262075Z ---------------------------------------------------------------------- 2022-03-17T20:35:38.3262411Z Ran 1 test in 5.286s 2022-03-17T20:35:38.3262524Z 2022-03-17T20:35:38.3262618Z OK 2022-03-17T20:35:38.3262765Z 2022-03-17T20:35:38.3262890Z Generating XML reports... 2022-03-17T20:35:38.3307034Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203533.xml 2022-03-17T20:35:39.6281419Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:39.6296616Z 2022-03-17T20:35:39.6296770Z Running tests... 2022-03-17T20:35:39.6297208Z ---------------------------------------------------------------------- 2022-03-17T20:35:41.1736620Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:35:41.2157246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5145 2022-03-17T20:35:41.2284901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5146 2022-03-17T20:35:42.2991009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:42.3269782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:42.3270580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:42.3297443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:42.3304474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:42.4285072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:45.0376366Z ok (5.408s) 2022-03-17T20:35:45.0376580Z 2022-03-17T20:35:45.0376958Z ---------------------------------------------------------------------- 2022-03-17T20:35:45.0377421Z Ran 1 test in 5.408s 2022-03-17T20:35:45.0377719Z 2022-03-17T20:35:45.0377817Z OK 2022-03-17T20:35:45.0377915Z 2022-03-17T20:35:45.0378032Z Generating XML reports... 2022-03-17T20:35:45.0421923Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203539.xml 2022-03-17T20:35:46.3511128Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:46.3526549Z 2022-03-17T20:35:46.3526693Z Running tests... 2022-03-17T20:35:46.3527466Z ---------------------------------------------------------------------- 2022-03-17T20:35:46.3547178Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:46.3547490Z 2022-03-17T20:35:46.3547796Z ---------------------------------------------------------------------- 2022-03-17T20:35:46.3548426Z Ran 1 test in 0.002s 2022-03-17T20:35:46.3548855Z 2022-03-17T20:35:46.3548983Z OK (skipped=1) 2022-03-17T20:35:46.3549142Z 2022-03-17T20:35:46.3549268Z Generating XML reports... 2022-03-17T20:35:46.3594332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203546.xml 2022-03-17T20:35:47.4914570Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:47.4929064Z 2022-03-17T20:35:47.4929201Z Running tests... 2022-03-17T20:35:47.4929904Z ---------------------------------------------------------------------- 2022-03-17T20:35:49.0164536Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:35:49.0597756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5303 2022-03-17T20:35:49.0725466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5304 2022-03-17T20:35:50.1398430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:50.1487868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:50.1489363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:50.1498810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:50.1505719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:50.1509224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:35:50.2502094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:50.2506641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:35:50.2507975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:35:50.2524019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:35:52.9817499Z ok (5.488s) 2022-03-17T20:35:52.9817707Z 2022-03-17T20:35:52.9818104Z ---------------------------------------------------------------------- 2022-03-17T20:35:52.9818455Z Ran 1 test in 5.489s 2022-03-17T20:35:52.9818619Z 2022-03-17T20:35:52.9818711Z OK 2022-03-17T20:35:52.9818853Z 2022-03-17T20:35:52.9818980Z Generating XML reports... 2022-03-17T20:35:52.9863913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203547.xml 2022-03-17T20:35:54.2918144Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:54.2933106Z 2022-03-17T20:35:54.2933438Z Running tests... 2022-03-17T20:35:54.2933879Z ---------------------------------------------------------------------- 2022-03-17T20:35:54.2953411Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:54.2953699Z 2022-03-17T20:35:54.2953962Z ---------------------------------------------------------------------- 2022-03-17T20:35:54.2954585Z Ran 1 test in 0.002s 2022-03-17T20:35:54.2954748Z 2022-03-17T20:35:54.2954858Z OK (skipped=1) 2022-03-17T20:35:54.2955017Z 2022-03-17T20:35:54.2955133Z Generating XML reports... 2022-03-17T20:35:54.2999676Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203554.xml 2022-03-17T20:35:55.4458907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:55.4474544Z 2022-03-17T20:35:55.4474790Z Running tests... 2022-03-17T20:35:55.4475277Z ---------------------------------------------------------------------- 2022-03-17T20:35:56.9969448Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:35:57.0394219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5465 2022-03-17T20:35:57.0521411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5466 2022-03-17T20:35:58.1089347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:35:58.1622093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:35:58.1623078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:58.1698769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:35:58.1705985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:35:58.2636579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:35:58.4567926Z skip (3.009s) 2022-03-17T20:35:58.4568224Z 2022-03-17T20:35:58.4568705Z ---------------------------------------------------------------------- 2022-03-17T20:35:58.4569024Z Ran 1 test in 3.009s 2022-03-17T20:35:58.4569211Z 2022-03-17T20:35:58.4569255Z OK (skipped=1) 2022-03-17T20:35:58.4569382Z 2022-03-17T20:35:58.4569493Z Generating XML reports... 2022-03-17T20:35:58.4615359Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203555.xml 2022-03-17T20:35:59.7432946Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:35:59.7448183Z 2022-03-17T20:35:59.7448635Z Running tests... 2022-03-17T20:35:59.7449112Z ---------------------------------------------------------------------- 2022-03-17T20:35:59.7468813Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:35:59.7469087Z 2022-03-17T20:35:59.7469371Z ---------------------------------------------------------------------- 2022-03-17T20:35:59.7469689Z Ran 1 test in 0.002s 2022-03-17T20:35:59.7469855Z 2022-03-17T20:35:59.7469981Z OK (skipped=1) 2022-03-17T20:35:59.7470107Z 2022-03-17T20:35:59.7470232Z Generating XML reports... 2022-03-17T20:35:59.7513777Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203559.xml 2022-03-17T20:36:00.8858162Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:00.8873721Z 2022-03-17T20:36:00.8873894Z Running tests... 2022-03-17T20:36:00.8874349Z ---------------------------------------------------------------------- 2022-03-17T20:36:00.8894287Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:00.8894592Z 2022-03-17T20:36:00.8894940Z ---------------------------------------------------------------------- 2022-03-17T20:36:00.8895267Z Ran 1 test in 0.002s 2022-03-17T20:36:00.8895414Z 2022-03-17T20:36:00.8895495Z OK (skipped=1) 2022-03-17T20:36:00.8895659Z 2022-03-17T20:36:00.8895786Z Generating XML reports... 2022-03-17T20:36:00.8939641Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203600.xml 2022-03-17T20:36:02.0411797Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:02.0427569Z 2022-03-17T20:36:02.0427978Z Running tests... 2022-03-17T20:36:02.0428477Z ---------------------------------------------------------------------- 2022-03-17T20:36:03.5811916Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:36:03.6236110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5648 2022-03-17T20:36:03.6362226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5649 2022-03-17T20:36:04.7330848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:04.7381923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:04.7382695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:04.7433237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:04.7440167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:04.8397651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:08.8476847Z ok (6.805s) 2022-03-17T20:36:08.8481258Z 2022-03-17T20:36:08.8481899Z ---------------------------------------------------------------------- 2022-03-17T20:36:08.8482248Z Ran 1 test in 6.805s 2022-03-17T20:36:08.8482422Z 2022-03-17T20:36:08.8482774Z OK 2022-03-17T20:36:08.8482972Z 2022-03-17T20:36:08.8483099Z Generating XML reports... 2022-03-17T20:36:08.8525199Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203602.xml 2022-03-17T20:36:10.1493851Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:10.1509754Z 2022-03-17T20:36:10.1510263Z Running tests... 2022-03-17T20:36:10.1510770Z ---------------------------------------------------------------------- 2022-03-17T20:36:11.7121392Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:36:11.7547066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5776 2022-03-17T20:36:11.7676229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5777 2022-03-17T20:36:12.8302809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:12.8639491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:12.8640281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:12.8707920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:12.8714666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:12.9654486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:16.9797073Z ok (6.828s) 2022-03-17T20:36:16.9797533Z 2022-03-17T20:36:16.9798231Z ---------------------------------------------------------------------- 2022-03-17T20:36:16.9798610Z Ran 1 test in 6.829s 2022-03-17T20:36:16.9798824Z 2022-03-17T20:36:16.9798917Z OK 2022-03-17T20:36:16.9799060Z 2022-03-17T20:36:16.9799174Z Generating XML reports... 2022-03-17T20:36:16.9846481Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203610.xml 2022-03-17T20:36:18.2825359Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:18.2840267Z 2022-03-17T20:36:18.2840395Z Running tests... 2022-03-17T20:36:18.2841306Z ---------------------------------------------------------------------- 2022-03-17T20:36:18.2860567Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:18.2861306Z 2022-03-17T20:36:18.2861867Z ---------------------------------------------------------------------- 2022-03-17T20:36:18.2862262Z Ran 1 test in 0.002s 2022-03-17T20:36:18.2862428Z 2022-03-17T20:36:18.2862537Z OK (skipped=1) 2022-03-17T20:36:18.2862693Z 2022-03-17T20:36:18.2863104Z Generating XML reports... 2022-03-17T20:36:18.2905877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203618.xml 2022-03-17T20:36:19.3885916Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:19.3901188Z 2022-03-17T20:36:19.3901433Z Running tests... 2022-03-17T20:36:19.3902042Z ---------------------------------------------------------------------- 2022-03-17T20:36:20.9537655Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:36:20.9969686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5939 2022-03-17T20:36:21.0100919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5940 2022-03-17T20:36:22.0520296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:22.1072066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:22.1073112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:22.1129345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:22.1136212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:22.1139307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:36:22.2086223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:22.2090792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:36:22.2091488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:36:22.2159082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:36:26.2217712Z ok (6.831s) 2022-03-17T20:36:26.2218685Z 2022-03-17T20:36:26.2219110Z ---------------------------------------------------------------------- 2022-03-17T20:36:26.2219488Z Ran 1 test in 6.832s 2022-03-17T20:36:26.2219734Z 2022-03-17T20:36:26.2219897Z OK 2022-03-17T20:36:26.2220097Z 2022-03-17T20:36:26.2220652Z Generating XML reports... 2022-03-17T20:36:26.2265492Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203619.xml 2022-03-17T20:36:27.5209489Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:27.5224629Z 2022-03-17T20:36:27.5224878Z Running tests... 2022-03-17T20:36:27.5225319Z ---------------------------------------------------------------------- 2022-03-17T20:36:27.5244665Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:27.5244979Z 2022-03-17T20:36:27.5245256Z ---------------------------------------------------------------------- 2022-03-17T20:36:27.5245580Z Ran 1 test in 0.002s 2022-03-17T20:36:27.5246032Z 2022-03-17T20:36:27.5246149Z OK (skipped=1) 2022-03-17T20:36:27.5246291Z 2022-03-17T20:36:27.5246419Z Generating XML reports... 2022-03-17T20:36:27.5290425Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203627.xml 2022-03-17T20:36:28.6705828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:28.6721087Z 2022-03-17T20:36:28.6721501Z Running tests... 2022-03-17T20:36:28.6722001Z ---------------------------------------------------------------------- 2022-03-17T20:36:30.2291907Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:36:30.2712412Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6106 2022-03-17T20:36:30.2839981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6107 2022-03-17T20:36:31.3253550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:31.3659083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:31.3659649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:31.3660529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:31.3666892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:31.3668991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:31.4882915Z skip (2.816s) 2022-03-17T20:36:31.4883121Z 2022-03-17T20:36:31.4883516Z ---------------------------------------------------------------------- 2022-03-17T20:36:31.4883851Z Ran 1 test in 2.816s 2022-03-17T20:36:31.4884019Z 2022-03-17T20:36:31.4884128Z OK (skipped=1) 2022-03-17T20:36:31.4884281Z 2022-03-17T20:36:31.4884408Z Generating XML reports... 2022-03-17T20:36:31.4930458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203628.xml 2022-03-17T20:36:32.7753564Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:32.7768278Z 2022-03-17T20:36:32.7768496Z Running tests... 2022-03-17T20:36:32.7768974Z ---------------------------------------------------------------------- 2022-03-17T20:36:32.7788861Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:32.7789148Z 2022-03-17T20:36:32.7789484Z ---------------------------------------------------------------------- 2022-03-17T20:36:32.7789793Z Ran 1 test in 0.002s 2022-03-17T20:36:32.7789961Z 2022-03-17T20:36:32.7790077Z OK (skipped=1) 2022-03-17T20:36:32.7790222Z 2022-03-17T20:36:32.7790300Z Generating XML reports... 2022-03-17T20:36:32.7834298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203632.xml 2022-03-17T20:36:33.9155122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:33.9169468Z 2022-03-17T20:36:33.9169708Z Running tests... 2022-03-17T20:36:33.9170127Z ---------------------------------------------------------------------- 2022-03-17T20:36:33.9190245Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:33.9190523Z 2022-03-17T20:36:33.9190813Z ---------------------------------------------------------------------- 2022-03-17T20:36:33.9191149Z Ran 1 test in 0.002s 2022-03-17T20:36:33.9191315Z 2022-03-17T20:36:33.9191428Z OK (skipped=1) 2022-03-17T20:36:33.9191593Z 2022-03-17T20:36:33.9191690Z Generating XML reports... 2022-03-17T20:36:33.9234590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203633.xml 2022-03-17T20:36:35.0597734Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:35.0612795Z 2022-03-17T20:36:35.0613201Z Running tests... 2022-03-17T20:36:35.0613676Z ---------------------------------------------------------------------- 2022-03-17T20:36:36.6162231Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:36:36.6588644Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6289 2022-03-17T20:36:36.6717239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6290 2022-03-17T20:36:37.7472220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:37.7563125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:37.7563892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:37.7573394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:37.7580035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:37.8578542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:40.4808395Z ok (5.419s) 2022-03-17T20:36:40.4808769Z 2022-03-17T20:36:40.4809386Z ---------------------------------------------------------------------- 2022-03-17T20:36:40.4809982Z Ran 1 test in 5.419s 2022-03-17T20:36:40.4810313Z 2022-03-17T20:36:40.4810484Z OK 2022-03-17T20:36:40.4810730Z 2022-03-17T20:36:40.4810961Z Generating XML reports... 2022-03-17T20:36:40.4856881Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203635.xml 2022-03-17T20:36:41.7741469Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:41.7756643Z 2022-03-17T20:36:41.7756881Z Running tests... 2022-03-17T20:36:41.7757343Z ---------------------------------------------------------------------- 2022-03-17T20:36:43.3017908Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:36:43.3442465Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6412 2022-03-17T20:36:43.3568665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6413 2022-03-17T20:36:44.4718982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:44.4747217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:44.4747988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:44.4820362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:44.4827915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:44.5762021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:47.1659870Z ok (5.390s) 2022-03-17T20:36:47.1660620Z 2022-03-17T20:36:47.1661237Z ---------------------------------------------------------------------- 2022-03-17T20:36:47.1661875Z Ran 1 test in 5.390s 2022-03-17T20:36:47.1662189Z 2022-03-17T20:36:47.1662354Z OK 2022-03-17T20:36:47.1662618Z 2022-03-17T20:36:47.1662855Z Generating XML reports... 2022-03-17T20:36:47.1710355Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203641.xml 2022-03-17T20:36:48.4659196Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:48.4674719Z 2022-03-17T20:36:48.4674845Z Running tests... 2022-03-17T20:36:48.4675310Z ---------------------------------------------------------------------- 2022-03-17T20:36:48.4694712Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:48.4694996Z 2022-03-17T20:36:48.4695282Z ---------------------------------------------------------------------- 2022-03-17T20:36:48.4695618Z Ran 1 test in 0.002s 2022-03-17T20:36:48.4695784Z 2022-03-17T20:36:48.4695875Z OK (skipped=1) 2022-03-17T20:36:48.4696033Z 2022-03-17T20:36:48.4696160Z Generating XML reports... 2022-03-17T20:36:48.4740267Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203648.xml 2022-03-17T20:36:49.6133144Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:49.6148804Z 2022-03-17T20:36:49.6149049Z Running tests... 2022-03-17T20:36:49.6149510Z ---------------------------------------------------------------------- 2022-03-17T20:36:51.1457266Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:36:51.1894632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6570 2022-03-17T20:36:51.2022202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6571 2022-03-17T20:36:52.2653383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:36:52.2706508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:36:52.2707337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:52.2755177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:36:52.2761721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:36:52.2766031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:36:52.3717076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:36:52.3720639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:36:52.3721331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:36:52.3783897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:36:55.0113587Z ok (5.396s) 2022-03-17T20:36:55.0113812Z 2022-03-17T20:36:55.0114194Z ---------------------------------------------------------------------- 2022-03-17T20:36:55.0114516Z Ran 1 test in 5.396s 2022-03-17T20:36:55.0114684Z 2022-03-17T20:36:55.0115252Z OK 2022-03-17T20:36:55.0115421Z 2022-03-17T20:36:55.0116004Z Generating XML reports... 2022-03-17T20:36:55.0160667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203649.xml 2022-03-17T20:36:56.3163527Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:56.3179446Z 2022-03-17T20:36:56.3179720Z Running tests... 2022-03-17T20:36:56.3180147Z ---------------------------------------------------------------------- 2022-03-17T20:36:56.3200223Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:36:56.3200838Z 2022-03-17T20:36:56.3201173Z ---------------------------------------------------------------------- 2022-03-17T20:36:56.3201512Z Ran 1 test in 0.002s 2022-03-17T20:36:56.3201677Z 2022-03-17T20:36:56.3201788Z OK (skipped=1) 2022-03-17T20:36:56.3202223Z 2022-03-17T20:36:56.3202328Z Generating XML reports... 2022-03-17T20:36:56.3246627Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203656.xml 2022-03-17T20:36:57.4496427Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:36:57.4511300Z 2022-03-17T20:36:57.4511533Z Running tests... 2022-03-17T20:36:57.4512025Z ---------------------------------------------------------------------- 2022-03-17T20:36:58.9662261Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:36:59.0080449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6732 2022-03-17T20:36:59.0206054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6733 2022-03-17T20:37:00.1001140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:00.1101060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:00.1101846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:00.1102550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:00.1110201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:00.1111285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:00.2248400Z skip (2.773s) 2022-03-17T20:37:00.2248669Z 2022-03-17T20:37:00.2249065Z ---------------------------------------------------------------------- 2022-03-17T20:37:00.2249415Z Ran 1 test in 2.774s 2022-03-17T20:37:00.2249580Z 2022-03-17T20:37:00.2249691Z OK (skipped=1) 2022-03-17T20:37:00.2249866Z 2022-03-17T20:37:00.2249920Z Generating XML reports... 2022-03-17T20:37:00.2294176Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203657.xml 2022-03-17T20:37:01.5073910Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:01.5089903Z 2022-03-17T20:37:01.5090055Z Running tests... 2022-03-17T20:37:01.5090910Z ---------------------------------------------------------------------- 2022-03-17T20:37:03.0332859Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:03.0755311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6845 2022-03-17T20:37:03.0880261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6846 2022-03-17T20:37:04.1315741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:04.1566883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:04.1567714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:04.1619713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:04.1627285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:04.2580627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:06.5553514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:37:06.5554096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:37:06.5554890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:37:06.5555860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:37:06.8972881Z ok (5.388s) 2022-03-17T20:37:06.8973100Z 2022-03-17T20:37:06.8973479Z ---------------------------------------------------------------------- 2022-03-17T20:37:06.8973793Z Ran 1 test in 5.388s 2022-03-17T20:37:06.8973959Z 2022-03-17T20:37:06.8974050Z OK 2022-03-17T20:37:06.8974190Z 2022-03-17T20:37:06.8974323Z Generating XML reports... 2022-03-17T20:37:06.9018776Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203701.xml 2022-03-17T20:37:08.1934780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:08.1950500Z 2022-03-17T20:37:08.1950741Z Running tests... 2022-03-17T20:37:08.1951138Z ---------------------------------------------------------------------- 2022-03-17T20:37:09.7290517Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:09.7710116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6979 2022-03-17T20:37:09.7836159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6980 2022-03-17T20:37:10.8189626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:10.8466828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:10.8468151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:10.8493786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:10.8501195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:10.9481693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:11.0883245Z skip (2.893s) 2022-03-17T20:37:11.0883487Z 2022-03-17T20:37:11.0883901Z ---------------------------------------------------------------------- 2022-03-17T20:37:11.0884232Z Ran 1 test in 2.893s 2022-03-17T20:37:11.0884397Z 2022-03-17T20:37:11.0884509Z OK (skipped=1) 2022-03-17T20:37:11.0884646Z 2022-03-17T20:37:11.0884771Z Generating XML reports... 2022-03-17T20:37:11.0930437Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203708.xml 2022-03-17T20:37:12.3741696Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:12.3756588Z 2022-03-17T20:37:12.3756886Z Running tests... 2022-03-17T20:37:12.3757352Z ---------------------------------------------------------------------- 2022-03-17T20:37:12.3777537Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:37:12.3777720Z 2022-03-17T20:37:12.3778060Z ---------------------------------------------------------------------- 2022-03-17T20:37:12.3778389Z Ran 1 test in 0.002s 2022-03-17T20:37:12.3778583Z 2022-03-17T20:37:12.3778660Z OK (skipped=1) 2022-03-17T20:37:12.3778825Z 2022-03-17T20:37:12.3778921Z Generating XML reports... 2022-03-17T20:37:12.3822531Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203712.xml 2022-03-17T20:37:13.5169663Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:13.5185434Z 2022-03-17T20:37:13.5185716Z Running tests... 2022-03-17T20:37:13.5186349Z ---------------------------------------------------------------------- 2022-03-17T20:37:13.5206167Z test_barrier (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:37:13.5206556Z 2022-03-17T20:37:13.5206960Z ---------------------------------------------------------------------- 2022-03-17T20:37:13.5207658Z Ran 1 test in 0.002s 2022-03-17T20:37:13.5207819Z 2022-03-17T20:37:13.5207948Z OK (skipped=1) 2022-03-17T20:37:13.5208066Z 2022-03-17T20:37:13.5208174Z Generating XML reports... 2022-03-17T20:37:13.5253584Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203713.xml 2022-03-17T20:37:14.6699743Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:14.6715665Z 2022-03-17T20:37:14.6715955Z Running tests... 2022-03-17T20:37:14.6716408Z ---------------------------------------------------------------------- 2022-03-17T20:37:16.2285294Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:16.2709222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7162 2022-03-17T20:37:16.2838838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7163 2022-03-17T20:37:17.3674326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:17.3704375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:17.3705140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:17.3775447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:17.3782137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:17.4719510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:20.7938632Z ok (6.122s) 2022-03-17T20:37:20.7938895Z 2022-03-17T20:37:20.7939325Z ---------------------------------------------------------------------- 2022-03-17T20:37:20.7939660Z Ran 1 test in 6.122s 2022-03-17T20:37:20.7939827Z 2022-03-17T20:37:20.7939919Z OK 2022-03-17T20:37:20.7940002Z 2022-03-17T20:37:20.7940128Z Generating XML reports... 2022-03-17T20:37:20.7985475Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203714.xml 2022-03-17T20:37:22.1003076Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:22.1017738Z 2022-03-17T20:37:22.1017976Z Running tests... 2022-03-17T20:37:22.1018396Z ---------------------------------------------------------------------- 2022-03-17T20:37:22.1038183Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:37:22.1038500Z 2022-03-17T20:37:22.1038999Z ---------------------------------------------------------------------- 2022-03-17T20:37:22.1039317Z Ran 1 test in 0.002s 2022-03-17T20:37:22.1039480Z 2022-03-17T20:37:22.1039587Z OK (skipped=1) 2022-03-17T20:37:22.1039771Z 2022-03-17T20:37:22.1039852Z Generating XML reports... 2022-03-17T20:37:22.1083264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203722.xml 2022-03-17T20:37:23.2446892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:23.2461837Z 2022-03-17T20:37:23.2462028Z Running tests... 2022-03-17T20:37:23.2462485Z ---------------------------------------------------------------------- 2022-03-17T20:37:24.8035348Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:24.8457991Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7320 2022-03-17T20:37:24.8586250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7321 2022-03-17T20:37:25.9251475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:25.9449952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:25.9451030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:25.9454143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:25.9461832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:26.0465006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:26.1633405Z skip (2.917s) 2022-03-17T20:37:26.1633812Z 2022-03-17T20:37:26.1634765Z ---------------------------------------------------------------------- 2022-03-17T20:37:26.1635394Z Ran 1 test in 2.917s 2022-03-17T20:37:26.1635763Z 2022-03-17T20:37:26.1635892Z OK (skipped=1) 2022-03-17T20:37:26.1636179Z 2022-03-17T20:37:26.1636387Z Generating XML reports... 2022-03-17T20:37:26.1681073Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203723.xml 2022-03-17T20:37:27.4418213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:27.4433442Z 2022-03-17T20:37:27.4433723Z Running tests... 2022-03-17T20:37:27.4434179Z ---------------------------------------------------------------------- 2022-03-17T20:37:27.4453768Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:37:27.4454053Z 2022-03-17T20:37:27.4454348Z ---------------------------------------------------------------------- 2022-03-17T20:37:27.4454694Z Ran 1 test in 0.002s 2022-03-17T20:37:27.4454822Z 2022-03-17T20:37:27.4454950Z OK (skipped=1) 2022-03-17T20:37:27.4455089Z 2022-03-17T20:37:27.4455198Z Generating XML reports... 2022-03-17T20:37:27.4499083Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203727.xml 2022-03-17T20:37:28.5989519Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:28.6005442Z 2022-03-17T20:37:28.6005707Z Running tests... 2022-03-17T20:37:28.6006134Z ---------------------------------------------------------------------- 2022-03-17T20:37:30.1355693Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:30.1794706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7468 2022-03-17T20:37:30.1927745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7469 2022-03-17T20:37:31.2614497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:31.3123105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:31.3123915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:31.3221989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:31.3229303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:31.4138057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:31.5974529Z skip (2.997s) 2022-03-17T20:37:31.5974743Z 2022-03-17T20:37:31.5975164Z ---------------------------------------------------------------------- 2022-03-17T20:37:31.5975420Z Ran 1 test in 2.997s 2022-03-17T20:37:31.5975599Z 2022-03-17T20:37:31.5975706Z OK (skipped=1) 2022-03-17T20:37:31.5975847Z 2022-03-17T20:37:31.5975985Z Generating XML reports... 2022-03-17T20:37:31.6020832Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203728.xml 2022-03-17T20:37:32.8822996Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:32.8838046Z 2022-03-17T20:37:32.8838632Z Running tests... 2022-03-17T20:37:32.8839231Z ---------------------------------------------------------------------- 2022-03-17T20:37:32.8859097Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:37:32.8859608Z 2022-03-17T20:37:32.8860116Z ---------------------------------------------------------------------- 2022-03-17T20:37:32.8860731Z Ran 1 test in 0.002s 2022-03-17T20:37:32.8860897Z 2022-03-17T20:37:32.8861007Z OK (skipped=1) 2022-03-17T20:37:32.8861235Z 2022-03-17T20:37:32.8861459Z Generating XML reports... 2022-03-17T20:37:32.8905648Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203732.xml 2022-03-17T20:37:34.0268640Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:34.0283797Z 2022-03-17T20:37:34.0283971Z Running tests... 2022-03-17T20:37:34.0284436Z ---------------------------------------------------------------------- 2022-03-17T20:37:34.0308930Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:37:34.0309406Z 2022-03-17T20:37:34.0309976Z ---------------------------------------------------------------------- 2022-03-17T20:37:34.0310665Z Ran 1 test in 0.003s 2022-03-17T20:37:34.0310859Z 2022-03-17T20:37:34.0310970Z OK (skipped=1) 2022-03-17T20:37:34.0311107Z 2022-03-17T20:37:34.0311240Z Generating XML reports... 2022-03-17T20:37:34.0354218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203734.xml 2022-03-17T20:37:35.1721751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:35.1736469Z 2022-03-17T20:37:35.1737005Z Running tests... 2022-03-17T20:37:35.1737632Z ---------------------------------------------------------------------- 2022-03-17T20:37:35.1758164Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:37:35.1758649Z 2022-03-17T20:37:35.1759278Z ---------------------------------------------------------------------- 2022-03-17T20:37:35.1759865Z Ran 1 test in 0.002s 2022-03-17T20:37:35.1760031Z 2022-03-17T20:37:35.1760138Z OK (skipped=1) 2022-03-17T20:37:35.1760295Z 2022-03-17T20:37:35.1760419Z Generating XML reports... 2022-03-17T20:37:35.1803234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203735.xml 2022-03-17T20:37:36.3192006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:36.3206975Z 2022-03-17T20:37:36.3207391Z Running tests... 2022-03-17T20:37:36.3207879Z ---------------------------------------------------------------------- 2022-03-17T20:37:36.3234200Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:37:36.3234489Z 2022-03-17T20:37:36.3234796Z ---------------------------------------------------------------------- 2022-03-17T20:37:36.3235101Z Ran 1 test in 0.003s 2022-03-17T20:37:36.3235265Z 2022-03-17T20:37:36.3235365Z OK (skipped=1) 2022-03-17T20:37:36.3235520Z 2022-03-17T20:37:36.3235648Z Generating XML reports... 2022-03-17T20:37:36.3279428Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203736.xml 2022-03-17T20:37:37.4788842Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:37.4803645Z 2022-03-17T20:37:37.4803889Z Running tests... 2022-03-17T20:37:37.4804331Z ---------------------------------------------------------------------- 2022-03-17T20:37:37.4831030Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:37:37.4831287Z 2022-03-17T20:37:37.4831580Z ---------------------------------------------------------------------- 2022-03-17T20:37:37.4832195Z Ran 1 test in 0.003s 2022-03-17T20:37:37.4832329Z 2022-03-17T20:37:37.4832444Z OK (skipped=1) 2022-03-17T20:37:37.4832599Z 2022-03-17T20:37:37.4832706Z Generating XML reports... 2022-03-17T20:37:37.4876835Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203737.xml 2022-03-17T20:37:38.6248180Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:38.6263577Z 2022-03-17T20:37:38.6263976Z Running tests... 2022-03-17T20:37:38.6264443Z ---------------------------------------------------------------------- 2022-03-17T20:37:40.1699351Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:40.2124945Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7756 2022-03-17T20:37:40.2251587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7757 2022-03-17T20:37:41.2704165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:41.3114128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:41.3114944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:41.3211342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:41.3218297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:41.4126722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:41.4353887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:37:41.4354444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:37:41.4355141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:37:41.4355837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:37:41.4357858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:37:41.4461046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:37:41.4461736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:37:41.4462383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:37:41.7301168Z ok (3.103s) 2022-03-17T20:37:41.7301451Z 2022-03-17T20:37:41.7301791Z ---------------------------------------------------------------------- 2022-03-17T20:37:41.7302127Z Ran 1 test in 3.104s 2022-03-17T20:37:41.7302292Z 2022-03-17T20:37:41.7302384Z OK 2022-03-17T20:37:41.7302523Z 2022-03-17T20:37:41.7302640Z Generating XML reports... 2022-03-17T20:37:41.7347478Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203738.xml 2022-03-17T20:37:43.0087130Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:43.0101913Z 2022-03-17T20:37:43.0102169Z Running tests... 2022-03-17T20:37:43.0102597Z ---------------------------------------------------------------------- 2022-03-17T20:37:44.5382568Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:44.5805470Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7879 2022-03-17T20:37:44.5930865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7880 2022-03-17T20:37:45.6330543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:45.6665066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:45.6665870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:45.6735439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:45.6742200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:45.7678054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:47.5007030Z ok (4.490s) 2022-03-17T20:37:47.5007281Z 2022-03-17T20:37:47.5007642Z ---------------------------------------------------------------------- 2022-03-17T20:37:47.5007957Z Ran 1 test in 4.490s 2022-03-17T20:37:47.5008148Z 2022-03-17T20:37:47.5008236Z OK 2022-03-17T20:37:47.5008370Z 2022-03-17T20:37:47.5008531Z Generating XML reports... 2022-03-17T20:37:47.5052881Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203743.xml 2022-03-17T20:37:48.8107655Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:48.8122645Z 2022-03-17T20:37:48.8123240Z Running tests... 2022-03-17T20:37:48.8123716Z ---------------------------------------------------------------------- 2022-03-17T20:37:50.3387181Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:50.3812048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8009 2022-03-17T20:37:50.3942435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8010 2022-03-17T20:37:51.4345471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:51.4840394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:51.4841223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:51.4851551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:51.4858166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:51.5855052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:51.6987557Z skip (2.886s) 2022-03-17T20:37:51.6987822Z 2022-03-17T20:37:51.6988155Z ---------------------------------------------------------------------- 2022-03-17T20:37:51.6988488Z Ran 1 test in 2.886s 2022-03-17T20:37:51.6988655Z 2022-03-17T20:37:51.6988794Z OK (skipped=1) 2022-03-17T20:37:51.6988888Z 2022-03-17T20:37:51.6989015Z Generating XML reports... 2022-03-17T20:37:51.7033877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203748.xml 2022-03-17T20:37:52.9879896Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:52.9894721Z 2022-03-17T20:37:52.9894951Z Running tests... 2022-03-17T20:37:52.9895399Z ---------------------------------------------------------------------- 2022-03-17T20:37:54.5397315Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:37:54.5821765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8122 2022-03-17T20:37:54.5947564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8123 2022-03-17T20:37:55.6609426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:37:55.7039580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:37:55.7040878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:55.7115429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:37:55.7121683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:37:55.8050899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:37:57.3018034Z ok (4.312s) 2022-03-17T20:37:57.3018306Z 2022-03-17T20:37:57.3018702Z ---------------------------------------------------------------------- 2022-03-17T20:37:57.3019044Z Ran 1 test in 4.312s 2022-03-17T20:37:57.3019186Z 2022-03-17T20:37:57.3019282Z OK 2022-03-17T20:37:57.3019417Z 2022-03-17T20:37:57.3019564Z Generating XML reports... 2022-03-17T20:37:57.3064398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203752.xml 2022-03-17T20:37:58.5897250Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:37:58.5912056Z 2022-03-17T20:37:58.5912474Z Running tests... 2022-03-17T20:37:58.5912964Z ---------------------------------------------------------------------- 2022-03-17T20:38:00.1185785Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:00.1608466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8236 2022-03-17T20:38:00.1735594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8237 2022-03-17T20:38:01.2158609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:01.2620825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:01.2621903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:01.2664554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:01.2672131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:01.3635216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:01.5782646Z ok (2.987s) 2022-03-17T20:38:01.5782912Z 2022-03-17T20:38:01.5783316Z ---------------------------------------------------------------------- 2022-03-17T20:38:01.5783654Z Ran 1 test in 2.987s 2022-03-17T20:38:01.5783796Z 2022-03-17T20:38:01.5783824Z OK 2022-03-17T20:38:01.5783958Z 2022-03-17T20:38:01.5784089Z Generating XML reports... 2022-03-17T20:38:01.5828356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203758.xml 2022-03-17T20:38:02.8763313Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:02.8778959Z 2022-03-17T20:38:02.8779433Z Running tests... 2022-03-17T20:38:02.8779921Z ---------------------------------------------------------------------- 2022-03-17T20:38:04.4274041Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:04.4703597Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8349 2022-03-17T20:38:04.4830184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8350 2022-03-17T20:38:05.5579061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:05.5723628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:05.5724671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:05.5782228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:05.5790146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:05.6736773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:07.2913750Z ok (4.413s) 2022-03-17T20:38:07.2913985Z 2022-03-17T20:38:07.2914750Z ---------------------------------------------------------------------- 2022-03-17T20:38:07.2915066Z Ran 1 test in 4.413s 2022-03-17T20:38:07.2915196Z 2022-03-17T20:38:07.2915261Z OK 2022-03-17T20:38:07.2915408Z 2022-03-17T20:38:07.2915542Z Generating XML reports... 2022-03-17T20:38:07.2960660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203802.xml 2022-03-17T20:38:08.5925292Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:08.5940595Z 2022-03-17T20:38:08.5940944Z Running tests... 2022-03-17T20:38:08.5941416Z ---------------------------------------------------------------------- 2022-03-17T20:38:10.1437476Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:10.1871700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8467 2022-03-17T20:38:10.2000106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8468 2022-03-17T20:38:11.2459035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:11.2514536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:11.2515723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:11.2561238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:11.2567502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:11.3528718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:11.6048167Z ok (3.010s) 2022-03-17T20:38:11.6048357Z 2022-03-17T20:38:11.6048703Z ---------------------------------------------------------------------- 2022-03-17T20:38:11.6049027Z Ran 1 test in 3.011s 2022-03-17T20:38:11.6049235Z 2022-03-17T20:38:11.6049308Z OK 2022-03-17T20:38:11.6049464Z 2022-03-17T20:38:11.6050088Z Generating XML reports... 2022-03-17T20:38:11.6094463Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203808.xml 2022-03-17T20:38:12.8825688Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:12.8840689Z 2022-03-17T20:38:12.8841166Z Running tests... 2022-03-17T20:38:12.8841610Z ---------------------------------------------------------------------- 2022-03-17T20:38:12.8861292Z test_broadcast (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:38:12.8861571Z 2022-03-17T20:38:12.8861833Z ---------------------------------------------------------------------- 2022-03-17T20:38:12.8862174Z Ran 1 test in 0.002s 2022-03-17T20:38:12.8862345Z 2022-03-17T20:38:12.8862430Z OK (skipped=1) 2022-03-17T20:38:12.8862586Z 2022-03-17T20:38:12.8862767Z Generating XML reports... 2022-03-17T20:38:12.8907711Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203812.xml 2022-03-17T20:38:14.0270990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:14.0286833Z 2022-03-17T20:38:14.0287088Z Running tests... 2022-03-17T20:38:14.0287531Z ---------------------------------------------------------------------- 2022-03-17T20:38:15.5577442Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:15.5997695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8615 2022-03-17T20:38:15.6127854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8616 2022-03-17T20:38:16.6529153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:16.6572086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:16.6572948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:16.6631950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:16.6638803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:16.7587474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:18.9207754Z ok (4.892s) 2022-03-17T20:38:18.9208058Z 2022-03-17T20:38:18.9208485Z ---------------------------------------------------------------------- 2022-03-17T20:38:18.9208797Z Ran 1 test in 4.892s 2022-03-17T20:38:18.9208938Z 2022-03-17T20:38:18.9209050Z OK 2022-03-17T20:38:18.9209169Z 2022-03-17T20:38:18.9209319Z Generating XML reports... 2022-03-17T20:38:18.9253631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203814.xml 2022-03-17T20:38:20.2178606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:20.2194539Z 2022-03-17T20:38:20.2194917Z Running tests... 2022-03-17T20:38:20.2195405Z ---------------------------------------------------------------------- 2022-03-17T20:38:20.2214943Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:38:20.2215610Z 2022-03-17T20:38:20.2215929Z ---------------------------------------------------------------------- 2022-03-17T20:38:20.2216236Z Ran 1 test in 0.002s 2022-03-17T20:38:20.2216437Z 2022-03-17T20:38:20.2216531Z OK (skipped=1) 2022-03-17T20:38:20.2216686Z 2022-03-17T20:38:20.2216841Z Generating XML reports... 2022-03-17T20:38:20.2260809Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203820.xml 2022-03-17T20:38:21.3686740Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:21.3702837Z 2022-03-17T20:38:21.3703212Z Running tests... 2022-03-17T20:38:21.3703665Z ---------------------------------------------------------------------- 2022-03-17T20:38:21.3723794Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:38:21.3724193Z 2022-03-17T20:38:21.3724490Z ---------------------------------------------------------------------- 2022-03-17T20:38:21.3724874Z Ran 1 test in 0.002s 2022-03-17T20:38:21.3725030Z 2022-03-17T20:38:21.3725116Z OK (skipped=1) 2022-03-17T20:38:21.3725268Z 2022-03-17T20:38:21.3725391Z Generating XML reports... 2022-03-17T20:38:21.3769647Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203821.xml 2022-03-17T20:38:22.5140843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:22.5156027Z 2022-03-17T20:38:22.5156153Z Running tests... 2022-03-17T20:38:22.5157002Z ---------------------------------------------------------------------- 2022-03-17T20:38:22.5178180Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:38:22.5178755Z 2022-03-17T20:38:22.5179318Z ---------------------------------------------------------------------- 2022-03-17T20:38:22.5179610Z Ran 1 test in 0.002s 2022-03-17T20:38:22.5179778Z 2022-03-17T20:38:22.5179924Z OK (skipped=1) 2022-03-17T20:38:22.5180045Z 2022-03-17T20:38:22.5180150Z Generating XML reports... 2022-03-17T20:38:22.5223904Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203822.xml 2022-03-17T20:38:23.6593532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:23.6608868Z 2022-03-17T20:38:23.6609364Z Running tests... 2022-03-17T20:38:23.6609788Z ---------------------------------------------------------------------- 2022-03-17T20:38:25.2241015Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:25.2665412Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8846 2022-03-17T20:38:25.2794333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8847 2022-03-17T20:38:26.3110414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:26.3363470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:26.3364526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:26.3413960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:26.3420812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:26.4378771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:28.9890375Z ok (5.328s) 2022-03-17T20:38:28.9890590Z 2022-03-17T20:38:28.9891028Z ---------------------------------------------------------------------- 2022-03-17T20:38:28.9891310Z Ran 1 test in 5.328s 2022-03-17T20:38:28.9891427Z 2022-03-17T20:38:28.9891517Z OK 2022-03-17T20:38:28.9891652Z 2022-03-17T20:38:28.9891766Z Generating XML reports... 2022-03-17T20:38:28.9935161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203823.xml 2022-03-17T20:38:30.2857304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:30.2872842Z 2022-03-17T20:38:30.2873128Z Running tests... 2022-03-17T20:38:30.2873614Z ---------------------------------------------------------------------- 2022-03-17T20:38:31.8386835Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:31.8812688Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8969 2022-03-17T20:38:31.8939300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8970 2022-03-17T20:38:32.9468940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:32.9585114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:32.9585936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:32.9671474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:32.9678197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:33.0597248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:33.0707332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:38:33.0707897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:38:33.0708842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:38:33.0709538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:38:33.0710345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:38:33.0710786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:38:33.0711562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:38:33.0712535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:38:34.7012238Z ok (4.414s) 2022-03-17T20:38:34.7012495Z 2022-03-17T20:38:34.7012951Z ---------------------------------------------------------------------- 2022-03-17T20:38:34.7013250Z Ran 1 test in 4.414s 2022-03-17T20:38:34.7013392Z 2022-03-17T20:38:34.7013482Z OK 2022-03-17T20:38:34.7013620Z 2022-03-17T20:38:34.7013825Z Generating XML reports... 2022-03-17T20:38:34.7057795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203830.xml 2022-03-17T20:38:35.9968333Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:35.9984020Z 2022-03-17T20:38:35.9984418Z Running tests... 2022-03-17T20:38:35.9984902Z ---------------------------------------------------------------------- 2022-03-17T20:38:37.5322225Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:37.5747689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9099 2022-03-17T20:38:37.5875578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9100 2022-03-17T20:38:38.6427905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:38.6501796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:38.6502598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:38.6529141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:38.6536139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:38.7515961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:38.7656453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:38:38.7656949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:38:38.7657655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:38:38.7658325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:38:38.7661151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:38:38.7661641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:38:38.7662334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:38:38.7663025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:38:40.4951749Z ok (4.496s) 2022-03-17T20:38:40.4951989Z 2022-03-17T20:38:40.4952444Z ---------------------------------------------------------------------- 2022-03-17T20:38:40.4952716Z Ran 1 test in 4.497s 2022-03-17T20:38:40.4952879Z 2022-03-17T20:38:40.4952968Z OK 2022-03-17T20:38:40.4953104Z 2022-03-17T20:38:40.4953240Z Generating XML reports... 2022-03-17T20:38:40.4997393Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203835.xml 2022-03-17T20:38:41.7990066Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:41.8004655Z 2022-03-17T20:38:41.8005019Z Running tests... 2022-03-17T20:38:41.8005841Z ---------------------------------------------------------------------- 2022-03-17T20:38:43.3355791Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:43.3784837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9229 2022-03-17T20:38:43.3911084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9230 2022-03-17T20:38:44.4699784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:44.4881449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:44.4882447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:44.4905450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:44.4912672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:44.5897193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:46.2103262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:46.2110639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:46.4989596Z ok (4.698s) 2022-03-17T20:38:46.4989900Z 2022-03-17T20:38:46.4990247Z ---------------------------------------------------------------------- 2022-03-17T20:38:46.4990591Z Ran 1 test in 4.699s 2022-03-17T20:38:46.4990760Z 2022-03-17T20:38:46.4992250Z OK 2022-03-17T20:38:46.4992492Z 2022-03-17T20:38:46.4992984Z Generating XML reports... 2022-03-17T20:38:46.5035569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203841.xml 2022-03-17T20:38:47.7970714Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:47.7985826Z 2022-03-17T20:38:47.7986169Z Running tests... 2022-03-17T20:38:47.7986701Z ---------------------------------------------------------------------- 2022-03-17T20:38:49.3421735Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:49.3859201Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9355 2022-03-17T20:38:49.3990576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9356 2022-03-17T20:38:50.4242492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:50.4655022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:50.4655834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:50.4749130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:50.4755700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:50.5669515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:52.1813009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:52.1813874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:52.1819283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:52.1822307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:52.5069709Z ok (4.708s) 2022-03-17T20:38:52.5069992Z 2022-03-17T20:38:52.5070357Z ---------------------------------------------------------------------- 2022-03-17T20:38:52.5070928Z Ran 1 test in 4.708s 2022-03-17T20:38:52.5071159Z 2022-03-17T20:38:52.5071256Z OK 2022-03-17T20:38:52.5071587Z 2022-03-17T20:38:52.5071701Z Generating XML reports... 2022-03-17T20:38:52.5116433Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203847.xml 2022-03-17T20:38:53.8100112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:53.8116619Z 2022-03-17T20:38:53.8116876Z Running tests... 2022-03-17T20:38:53.8117340Z ---------------------------------------------------------------------- 2022-03-17T20:38:55.3550856Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:38:55.3976622Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9481 2022-03-17T20:38:55.4103643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9482 2022-03-17T20:38:56.4711188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:38:56.4989988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:38:56.4991429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:56.5015242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:38:56.5021662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:38:56.6006141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:38:58.2139937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:58.2141100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:58.2149696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:58.2152022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:58.2268815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:58.2269328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:58.2276696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:58.2278969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:38:58.5182239Z ok (4.706s) 2022-03-17T20:38:58.5182457Z 2022-03-17T20:38:58.5182978Z ---------------------------------------------------------------------- 2022-03-17T20:38:58.5183305Z Ran 1 test in 4.706s 2022-03-17T20:38:58.5183470Z 2022-03-17T20:38:58.5183578Z OK 2022-03-17T20:38:58.5183712Z 2022-03-17T20:38:58.5183847Z Generating XML reports... 2022-03-17T20:38:58.5227967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203853.xml 2022-03-17T20:38:59.8174143Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:38:59.8190034Z 2022-03-17T20:38:59.8190564Z Running tests... 2022-03-17T20:38:59.8191058Z ---------------------------------------------------------------------- 2022-03-17T20:39:01.3566438Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:01.3992786Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9607 2022-03-17T20:39:01.4122377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9608 2022-03-17T20:39:02.5006686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:02.5079201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:02.5080024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:02.5108428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:02.5114850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:02.6094090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:04.2429650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:04.2430178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:04.2437262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:04.2437779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:04.2543728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:04.2544226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:04.2551759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:04.2552197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:04.5200137Z ok (4.701s) 2022-03-17T20:39:04.5200429Z 2022-03-17T20:39:04.5200911Z ---------------------------------------------------------------------- 2022-03-17T20:39:04.5201230Z Ran 1 test in 4.701s 2022-03-17T20:39:04.5201396Z 2022-03-17T20:39:04.5201488Z OK 2022-03-17T20:39:04.5201630Z 2022-03-17T20:39:04.5201762Z Generating XML reports... 2022-03-17T20:39:04.5246116Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203859.xml 2022-03-17T20:39:05.8172954Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:05.8189014Z 2022-03-17T20:39:05.8189375Z Running tests... 2022-03-17T20:39:05.8189807Z ---------------------------------------------------------------------- 2022-03-17T20:39:07.3499601Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:07.3922488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9733 2022-03-17T20:39:07.4048256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9734 2022-03-17T20:39:08.4467180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:08.4776424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:08.4777408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:08.4873805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:08.4880351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:08.5791393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:10.8652762Z 2022-03-17T20:39:11.2154796Z ok (5.396s) 2022-03-17T20:39:11.2155099Z 2022-03-17T20:39:11.2155557Z ---------------------------------------------------------------------- 2022-03-17T20:39:11.2155877Z Ran 1 test in 5.397s 2022-03-17T20:39:11.2156021Z 2022-03-17T20:39:11.2156114Z OK 2022-03-17T20:39:11.2156257Z 2022-03-17T20:39:11.2156385Z Generating XML reports... 2022-03-17T20:39:11.2200579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203905.xml 2022-03-17T20:39:12.5101010Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:12.5115714Z 2022-03-17T20:39:12.5115951Z Running tests... 2022-03-17T20:39:12.5116528Z ---------------------------------------------------------------------- 2022-03-17T20:39:14.0573074Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:14.0997382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9856 2022-03-17T20:39:14.1123762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9857 2022-03-17T20:39:15.1444872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:15.1946544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:15.1947638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:15.1951296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:15.1958533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:15.2962665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:17.9216104Z ok (5.410s) 2022-03-17T20:39:17.9216317Z 2022-03-17T20:39:17.9216733Z ---------------------------------------------------------------------- 2022-03-17T20:39:17.9217074Z Ran 1 test in 5.410s 2022-03-17T20:39:17.9217244Z 2022-03-17T20:39:17.9217341Z OK 2022-03-17T20:39:17.9217481Z 2022-03-17T20:39:17.9217596Z Generating XML reports... 2022-03-17T20:39:17.9264775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203912.xml 2022-03-17T20:39:19.2158472Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:19.2175851Z 2022-03-17T20:39:19.2177912Z Running tests... 2022-03-17T20:39:19.2178648Z ---------------------------------------------------------------------- 2022-03-17T20:39:20.7459402Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:20.7886468Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9979 2022-03-17T20:39:20.8015436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9980 2022-03-17T20:39:21.8392442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:21.8777659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:21.8778487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:21.8797659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:21.8804304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:21.9792696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:24.5698161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:24.5698678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:24.9133790Z ok (5.696s) 2022-03-17T20:39:24.9134023Z 2022-03-17T20:39:24.9134427Z ---------------------------------------------------------------------- 2022-03-17T20:39:24.9134766Z Ran 1 test in 5.696s 2022-03-17T20:39:24.9134914Z 2022-03-17T20:39:24.9134962Z OK 2022-03-17T20:39:24.9135101Z 2022-03-17T20:39:24.9135231Z Generating XML reports... 2022-03-17T20:39:24.9182421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203919.xml 2022-03-17T20:39:26.1643148Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:26.1658252Z 2022-03-17T20:39:26.1658401Z Running tests... 2022-03-17T20:39:26.1659232Z ---------------------------------------------------------------------- 2022-03-17T20:39:27.7127708Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:27.7552711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10106 2022-03-17T20:39:27.7680093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10107 2022-03-17T20:39:28.8246614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:28.8395256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:28.8395983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:28.8449710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:28.8456373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:28.9409500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:30.5495552Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:39:30.8759103Z ok (4.710s) 2022-03-17T20:39:30.8759310Z 2022-03-17T20:39:30.8760075Z ---------------------------------------------------------------------- 2022-03-17T20:39:30.8760666Z Ran 1 test in 4.710s 2022-03-17T20:39:30.8760818Z 2022-03-17T20:39:30.8760909Z OK 2022-03-17T20:39:30.8761045Z 2022-03-17T20:39:30.8761176Z Generating XML reports... 2022-03-17T20:39:30.8805831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203926.xml 2022-03-17T20:39:32.1758549Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:32.1773474Z 2022-03-17T20:39:32.1773908Z Running tests... 2022-03-17T20:39:32.1774336Z ---------------------------------------------------------------------- 2022-03-17T20:39:33.7054722Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:33.7478139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10232 2022-03-17T20:39:33.7605112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10233 2022-03-17T20:39:34.8148881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:34.8174467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:34.8175582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:34.8251570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:34.8257643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:34.9190133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:36.5472995Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:39:36.5474651Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:39:36.8684507Z ok (4.691s) 2022-03-17T20:39:36.8684914Z 2022-03-17T20:39:36.8685578Z ---------------------------------------------------------------------- 2022-03-17T20:39:36.8685955Z Ran 1 test in 4.691s 2022-03-17T20:39:36.8686128Z 2022-03-17T20:39:36.8686186Z OK 2022-03-17T20:39:36.8686264Z 2022-03-17T20:39:36.8686397Z Generating XML reports... 2022-03-17T20:39:36.8731930Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203932.xml 2022-03-17T20:39:38.1713641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:38.1728680Z 2022-03-17T20:39:38.1729124Z Running tests... 2022-03-17T20:39:38.1729576Z ---------------------------------------------------------------------- 2022-03-17T20:39:38.1754504Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:39:38.1754987Z 2022-03-17T20:39:38.1755362Z ---------------------------------------------------------------------- 2022-03-17T20:39:38.1755694Z Ran 1 test in 0.003s 2022-03-17T20:39:38.1755823Z 2022-03-17T20:39:38.1755948Z OK (skipped=1) 2022-03-17T20:39:38.1756102Z 2022-03-17T20:39:38.1756225Z Generating XML reports... 2022-03-17T20:39:38.1799998Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203938.xml 2022-03-17T20:39:39.3270411Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:39.3285747Z 2022-03-17T20:39:39.3286058Z Running tests... 2022-03-17T20:39:39.3286494Z ---------------------------------------------------------------------- 2022-03-17T20:39:40.8836205Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:40.9257578Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10393 2022-03-17T20:39:40.9384453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10394 2022-03-17T20:39:41.9830132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:42.0300573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:42.0301642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:42.0336948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:42.0343871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:42.1316196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:44.7527359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:44.7527910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:45.1500053Z ok (5.821s) 2022-03-17T20:39:45.1500255Z 2022-03-17T20:39:45.1501024Z ---------------------------------------------------------------------- 2022-03-17T20:39:45.1501378Z Ran 1 test in 5.821s 2022-03-17T20:39:45.1501548Z 2022-03-17T20:39:45.1501635Z OK 2022-03-17T20:39:45.1501712Z 2022-03-17T20:39:45.1501814Z Generating XML reports... 2022-03-17T20:39:45.1547179Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203939.xml 2022-03-17T20:39:46.4485690Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:46.4500741Z 2022-03-17T20:39:46.4501090Z Running tests... 2022-03-17T20:39:46.4501482Z ---------------------------------------------------------------------- 2022-03-17T20:39:47.9871380Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:48.0296380Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10520 2022-03-17T20:39:48.0424611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10521 2022-03-17T20:39:49.1043448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:49.1482485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:49.1483834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:49.1549567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:49.1556050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:49.2497268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:51.5438801Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T20:39:51.5440514Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T20:39:51.5442668Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T20:39:51.5444321Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T20:39:52.1521948Z ok (5.702s) 2022-03-17T20:39:52.1522129Z 2022-03-17T20:39:52.1522577Z ---------------------------------------------------------------------- 2022-03-17T20:39:52.1522899Z Ran 1 test in 5.702s 2022-03-17T20:39:52.1523263Z 2022-03-17T20:39:52.1523356Z OK 2022-03-17T20:39:52.1523513Z 2022-03-17T20:39:52.1523642Z Generating XML reports... 2022-03-17T20:39:52.1568308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203946.xml 2022-03-17T20:39:53.4543077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:39:53.4558896Z 2022-03-17T20:39:53.4559111Z Running tests... 2022-03-17T20:39:53.4559567Z ---------------------------------------------------------------------- 2022-03-17T20:39:55.0107351Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:39:55.0549634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10677 2022-03-17T20:39:55.0679882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10678 2022-03-17T20:39:56.1135083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:39:56.1300711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:39:56.1302193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:56.1337435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:39:56.1344876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:39:56.2315441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:39:59.8577403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:39:59.8577935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:01.1808082Z ok (7.724s) 2022-03-17T20:40:01.1808398Z 2022-03-17T20:40:01.1808720Z ---------------------------------------------------------------------- 2022-03-17T20:40:01.1809053Z Ran 1 test in 7.725s 2022-03-17T20:40:01.1809217Z 2022-03-17T20:40:01.1809308Z OK 2022-03-17T20:40:01.1809442Z 2022-03-17T20:40:01.1809595Z Generating XML reports... 2022-03-17T20:40:01.1854666Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203953.xml 2022-03-17T20:40:02.5229274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:02.5245646Z 2022-03-17T20:40:02.5246030Z Running tests... 2022-03-17T20:40:02.5246564Z ---------------------------------------------------------------------- 2022-03-17T20:40:04.0861619Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:40:04.1291218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10803 2022-03-17T20:40:04.1425260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10804 2022-03-17T20:40:05.2089236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:05.2247365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:05.2248200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:05.2291771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:05.2298334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:05.3262381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:07.8863114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:07.8863928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:08.2521794Z ok (5.727s) 2022-03-17T20:40:08.2522118Z 2022-03-17T20:40:08.2522669Z ---------------------------------------------------------------------- 2022-03-17T20:40:08.2523130Z Ran 1 test in 5.728s 2022-03-17T20:40:08.2523282Z 2022-03-17T20:40:08.2523387Z OK 2022-03-17T20:40:08.2523513Z 2022-03-17T20:40:08.2523640Z Generating XML reports... 2022-03-17T20:40:08.2569089Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204002.xml 2022-03-17T20:40:09.5540751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:09.5556631Z 2022-03-17T20:40:09.5556884Z Running tests... 2022-03-17T20:40:09.5557335Z ---------------------------------------------------------------------- 2022-03-17T20:40:11.1084544Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:40:11.1513989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10930 2022-03-17T20:40:11.1652234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10931 2022-03-17T20:40:12.2381642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:12.2707334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:12.2708155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:12.2787227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:12.2794051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:12.3721991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:14.9782673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:14.9783201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:14.9791605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:14.9792498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:15.4749902Z ok (5.919s) 2022-03-17T20:40:15.4750075Z 2022-03-17T20:40:15.4750433Z ---------------------------------------------------------------------- 2022-03-17T20:40:15.4750830Z Ran 1 test in 5.919s 2022-03-17T20:40:15.4750987Z 2022-03-17T20:40:15.4751079Z OK 2022-03-17T20:40:15.4751204Z 2022-03-17T20:40:15.4751296Z Generating XML reports... 2022-03-17T20:40:15.4796386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204009.xml 2022-03-17T20:40:16.7712838Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:16.7729042Z 2022-03-17T20:40:16.7729488Z Running tests... 2022-03-17T20:40:16.7729985Z ---------------------------------------------------------------------- 2022-03-17T20:40:18.3256998Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:40:18.3684771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11057 2022-03-17T20:40:18.3813292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11058 2022-03-17T20:40:19.4493687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:19.4835630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:19.4836418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:19.4899095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:19.4906034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:19.4909552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:40:19.5848360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:19.5853981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:40:19.5854987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:40:19.5928926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:40:22.1930239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:22.1930770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:22.1939788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:22.1941178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:22.6914938Z ok (5.918s) 2022-03-17T20:40:22.6915289Z 2022-03-17T20:40:22.6915958Z ---------------------------------------------------------------------- 2022-03-17T20:40:22.6916266Z Ran 1 test in 5.918s 2022-03-17T20:40:22.6916434Z 2022-03-17T20:40:22.6916524Z OK 2022-03-17T20:40:22.6916667Z 2022-03-17T20:40:22.6917428Z Generating XML reports... 2022-03-17T20:40:22.6957759Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204016.xml 2022-03-17T20:40:23.9783798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:23.9798953Z 2022-03-17T20:40:23.9799100Z Running tests... 2022-03-17T20:40:23.9799553Z ---------------------------------------------------------------------- 2022-03-17T20:40:25.5306543Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:40:25.5731356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11188 2022-03-17T20:40:25.5857304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11189 2022-03-17T20:40:26.6287427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:26.6710424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:26.6711230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:26.6796073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:26.6802695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:26.6804703Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-03-17T20:40:26.7725637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:26.7726387Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-03-17T20:40:29.3880601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:29.3881112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:29.3890514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:29.3891292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:29.4093321Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-03-17T20:40:29.4095606Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-03-17T20:40:29.6162527Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-03-17T20:40:29.6164746Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-03-17T20:40:29.6219917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:29.6220660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:29.6232187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:29.6232692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:30.1961357Z ok (6.216s) 2022-03-17T20:40:30.1961549Z 2022-03-17T20:40:30.1961954Z ---------------------------------------------------------------------- 2022-03-17T20:40:30.1962305Z Ran 1 test in 6.216s 2022-03-17T20:40:30.1962456Z 2022-03-17T20:40:30.1962542Z OK 2022-03-17T20:40:30.1962679Z 2022-03-17T20:40:30.1962787Z Generating XML reports... 2022-03-17T20:40:30.2008685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204023.xml 2022-03-17T20:40:31.4918592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:31.4933143Z 2022-03-17T20:40:31.4933291Z Running tests... 2022-03-17T20:40:31.4933996Z ---------------------------------------------------------------------- 2022-03-17T20:40:33.0331290Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:40:33.0755167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11315 2022-03-17T20:40:33.0879128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11316 2022-03-17T20:40:34.1488170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:34.1636454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:34.1637580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:34.1690814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:34.1697473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:34.1700071Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:40:34.2651759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:34.2653108Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:40:36.8638998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:36.8639549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:36.8648479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:36.8649289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:36.8655233Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T20:40:36.8655787Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T20:40:36.8681293Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T20:40:36.8681977Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T20:40:36.8683258Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T20:40:36.8683933Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T20:40:36.8685391Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-03-17T20:40:36.8686025Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-03-17T20:40:37.1512684Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:40:37.1515900Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:40:37.1573022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:37.1573478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:37.1584524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:37.1585486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:37.1590091Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T20:40:37.1591927Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T20:40:37.1615309Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T20:40:37.1617306Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T20:40:37.1617981Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T20:40:37.1619355Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T20:40:37.7984474Z ok (6.305s) 2022-03-17T20:40:37.7984831Z 2022-03-17T20:40:37.7985431Z ---------------------------------------------------------------------- 2022-03-17T20:40:37.7985774Z Ran 1 test in 6.305s 2022-03-17T20:40:37.7985895Z 2022-03-17T20:40:37.7985989Z OK 2022-03-17T20:40:37.7986135Z 2022-03-17T20:40:37.7986272Z Generating XML reports... 2022-03-17T20:40:37.8031062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204031.xml 2022-03-17T20:40:39.0863653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:39.0879536Z 2022-03-17T20:40:39.0879685Z Running tests... 2022-03-17T20:40:39.0880494Z ---------------------------------------------------------------------- 2022-03-17T20:40:39.0903798Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:39.0904232Z 2022-03-17T20:40:39.0904534Z ---------------------------------------------------------------------- 2022-03-17T20:40:39.0904863Z Ran 1 test in 0.002s 2022-03-17T20:40:39.0904987Z 2022-03-17T20:40:39.0905077Z OK (skipped=1) 2022-03-17T20:40:39.0905508Z 2022-03-17T20:40:39.0905647Z Generating XML reports... 2022-03-17T20:40:39.0949209Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204039.xml 2022-03-17T20:40:40.2325200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:40.2340427Z 2022-03-17T20:40:40.2340719Z Running tests... 2022-03-17T20:40:40.2341443Z ---------------------------------------------------------------------- 2022-03-17T20:40:40.2364121Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:40.2364451Z 2022-03-17T20:40:40.2364765Z ---------------------------------------------------------------------- 2022-03-17T20:40:40.2365085Z Ran 1 test in 0.002s 2022-03-17T20:40:40.2365242Z 2022-03-17T20:40:40.2365307Z OK (skipped=1) 2022-03-17T20:40:40.2365459Z 2022-03-17T20:40:40.2365565Z Generating XML reports... 2022-03-17T20:40:40.2409254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204040.xml 2022-03-17T20:40:41.3876654Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:41.3892276Z 2022-03-17T20:40:41.3893419Z Running tests... 2022-03-17T20:40:41.3893931Z ---------------------------------------------------------------------- 2022-03-17T20:40:41.3917688Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:41.3918074Z 2022-03-17T20:40:41.3918403Z ---------------------------------------------------------------------- 2022-03-17T20:40:41.3918697Z Ran 1 test in 0.003s 2022-03-17T20:40:41.3918866Z 2022-03-17T20:40:41.3918964Z OK (skipped=1) 2022-03-17T20:40:41.3919095Z 2022-03-17T20:40:41.3919218Z Generating XML reports... 2022-03-17T20:40:41.3964077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204041.xml 2022-03-17T20:40:42.5299801Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:42.5314567Z 2022-03-17T20:40:42.5314742Z Running tests... 2022-03-17T20:40:42.5315181Z ---------------------------------------------------------------------- 2022-03-17T20:40:42.5339196Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:42.5339624Z 2022-03-17T20:40:42.5339905Z ---------------------------------------------------------------------- 2022-03-17T20:40:42.5340225Z Ran 1 test in 0.002s 2022-03-17T20:40:42.5340599Z 2022-03-17T20:40:42.5340715Z OK (skipped=1) 2022-03-17T20:40:42.5340870Z 2022-03-17T20:40:42.5340974Z Generating XML reports... 2022-03-17T20:40:42.5384033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204042.xml 2022-03-17T20:40:43.6836477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:43.6851860Z 2022-03-17T20:40:43.6851998Z Running tests... 2022-03-17T20:40:43.6852778Z ---------------------------------------------------------------------- 2022-03-17T20:40:43.6877681Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:43.6878010Z 2022-03-17T20:40:43.6878320Z ---------------------------------------------------------------------- 2022-03-17T20:40:43.6878657Z Ran 1 test in 0.003s 2022-03-17T20:40:43.6878829Z 2022-03-17T20:40:43.6878936Z OK (skipped=1) 2022-03-17T20:40:43.6879097Z 2022-03-17T20:40:43.6881972Z Generating XML reports... 2022-03-17T20:40:43.6924392Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204043.xml 2022-03-17T20:40:44.8396080Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:44.8411278Z 2022-03-17T20:40:44.8411589Z Running tests... 2022-03-17T20:40:44.8412040Z ---------------------------------------------------------------------- 2022-03-17T20:40:44.8436395Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:44.8436786Z 2022-03-17T20:40:44.8437249Z ---------------------------------------------------------------------- 2022-03-17T20:40:44.8437704Z Ran 1 test in 0.003s 2022-03-17T20:40:44.8437866Z 2022-03-17T20:40:44.8437960Z OK (skipped=1) 2022-03-17T20:40:44.8438118Z 2022-03-17T20:40:44.8438244Z Generating XML reports... 2022-03-17T20:40:44.8482326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204044.xml 2022-03-17T20:40:45.9933692Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:45.9949235Z 2022-03-17T20:40:45.9949624Z Running tests... 2022-03-17T20:40:45.9950126Z ---------------------------------------------------------------------- 2022-03-17T20:40:45.9973757Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:45.9974075Z 2022-03-17T20:40:45.9974415Z ---------------------------------------------------------------------- 2022-03-17T20:40:45.9974786Z Ran 1 test in 0.002s 2022-03-17T20:40:45.9974936Z 2022-03-17T20:40:45.9975041Z OK (skipped=1) 2022-03-17T20:40:45.9975148Z 2022-03-17T20:40:45.9975281Z Generating XML reports... 2022-03-17T20:40:46.0018841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204045.xml 2022-03-17T20:40:47.1289188Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:47.1303606Z 2022-03-17T20:40:47.1304083Z Running tests... 2022-03-17T20:40:47.1304779Z ---------------------------------------------------------------------- 2022-03-17T20:40:47.1328438Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:47.1328884Z 2022-03-17T20:40:47.1329151Z ---------------------------------------------------------------------- 2022-03-17T20:40:47.1329642Z Ran 1 test in 0.002s 2022-03-17T20:40:47.1329882Z 2022-03-17T20:40:47.1329975Z OK (skipped=1) 2022-03-17T20:40:47.1330129Z 2022-03-17T20:40:47.1330252Z Generating XML reports... 2022-03-17T20:40:47.1371584Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204047.xml 2022-03-17T20:40:48.2812689Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:48.2828084Z 2022-03-17T20:40:48.2828514Z Running tests... 2022-03-17T20:40:48.2829017Z ---------------------------------------------------------------------- 2022-03-17T20:40:48.2852770Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:48.2853507Z 2022-03-17T20:40:48.2853757Z ---------------------------------------------------------------------- 2022-03-17T20:40:48.2854083Z Ran 1 test in 0.002s 2022-03-17T20:40:48.2854228Z 2022-03-17T20:40:48.2854335Z OK (skipped=1) 2022-03-17T20:40:48.2854507Z 2022-03-17T20:40:48.2854602Z Generating XML reports... 2022-03-17T20:40:48.2897616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204048.xml 2022-03-17T20:40:49.4333601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:49.4349264Z 2022-03-17T20:40:49.4349440Z Running tests... 2022-03-17T20:40:49.4349868Z ---------------------------------------------------------------------- 2022-03-17T20:40:49.4374143Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:49.4374552Z 2022-03-17T20:40:49.4374817Z ---------------------------------------------------------------------- 2022-03-17T20:40:49.4375116Z Ran 1 test in 0.003s 2022-03-17T20:40:49.4375279Z 2022-03-17T20:40:49.4375388Z OK (skipped=1) 2022-03-17T20:40:49.4375543Z 2022-03-17T20:40:49.4375668Z Generating XML reports... 2022-03-17T20:40:49.4418613Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204049.xml 2022-03-17T20:40:50.5873878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:50.5888700Z 2022-03-17T20:40:50.5889099Z Running tests... 2022-03-17T20:40:50.5889621Z ---------------------------------------------------------------------- 2022-03-17T20:40:50.5912174Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:50.5912443Z 2022-03-17T20:40:50.5912905Z ---------------------------------------------------------------------- 2022-03-17T20:40:50.5913325Z Ran 1 test in 0.002s 2022-03-17T20:40:50.5913491Z 2022-03-17T20:40:50.5913557Z OK (skipped=1) 2022-03-17T20:40:50.5913715Z 2022-03-17T20:40:50.5913838Z Generating XML reports... 2022-03-17T20:40:50.5957344Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204050.xml 2022-03-17T20:40:51.7375554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:51.7390772Z 2022-03-17T20:40:51.7391138Z Running tests... 2022-03-17T20:40:51.7391663Z ---------------------------------------------------------------------- 2022-03-17T20:40:51.7413584Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:40:51.7414023Z 2022-03-17T20:40:51.7414591Z ---------------------------------------------------------------------- 2022-03-17T20:40:51.7415212Z Ran 1 test in 0.002s 2022-03-17T20:40:51.7415379Z 2022-03-17T20:40:51.7415488Z OK (skipped=1) 2022-03-17T20:40:51.7415624Z 2022-03-17T20:40:51.7415743Z Generating XML reports... 2022-03-17T20:40:51.7459385Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204051.xml 2022-03-17T20:40:52.8949762Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:52.8964656Z 2022-03-17T20:40:52.8965168Z Running tests... 2022-03-17T20:40:52.8965639Z ---------------------------------------------------------------------- 2022-03-17T20:40:54.4406557Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:40:54.4830309Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11862 2022-03-17T20:40:54.4965102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11863 2022-03-17T20:40:55.5389469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:40:55.5426841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:40:55.5427629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:55.5490959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:40:55.5497552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:40:55.6441931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:40:58.1713282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.1713778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.1848101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.1848595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.1962076Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:40:58.1963811Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:40:58.2287718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.2288229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.2422087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.2422588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.2512504Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:40:58.2513375Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:40:58.2514515Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:40:58.2515338Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:40:58.2560587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.2561084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.2693046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.2693681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:40:58.6050748Z ok (5.708s) 2022-03-17T20:40:58.6051015Z 2022-03-17T20:40:58.6051470Z ---------------------------------------------------------------------- 2022-03-17T20:40:58.6051771Z Ran 1 test in 5.709s 2022-03-17T20:40:58.6051938Z 2022-03-17T20:40:58.6052101Z OK 2022-03-17T20:40:58.6052332Z 2022-03-17T20:40:58.6052529Z Generating XML reports... 2022-03-17T20:40:58.6097598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204052.xml 2022-03-17T20:40:59.9290229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:40:59.9306177Z 2022-03-17T20:40:59.9306600Z Running tests... 2022-03-17T20:40:59.9307082Z ---------------------------------------------------------------------- 2022-03-17T20:41:01.4653368Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:41:01.5104157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11989 2022-03-17T20:41:01.5234659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11990 2022-03-17T20:41:02.5744320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:02.6015397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:02.6016250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:02.6050634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:02.6057352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:02.7030498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:05.1324162Z ok (5.201s) 2022-03-17T20:41:05.1324417Z 2022-03-17T20:41:05.1324821Z ---------------------------------------------------------------------- 2022-03-17T20:41:05.1325135Z Ran 1 test in 5.202s 2022-03-17T20:41:05.1325332Z 2022-03-17T20:41:05.1325436Z OK 2022-03-17T20:41:05.1325571Z 2022-03-17T20:41:05.1325687Z Generating XML reports... 2022-03-17T20:41:05.1371036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204059.xml 2022-03-17T20:41:06.4298518Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:06.4314171Z 2022-03-17T20:41:06.4314790Z Running tests... 2022-03-17T20:41:06.4315364Z ---------------------------------------------------------------------- 2022-03-17T20:41:07.9840765Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:41:08.0263286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12111 2022-03-17T20:41:08.0389713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12112 2022-03-17T20:41:09.0794916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:09.0860602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:09.0861732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:09.0896000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:09.0903107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:09.1875724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:11.8041052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:41:11.8041557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:41:12.1487412Z ok (5.717s) 2022-03-17T20:41:12.1487635Z 2022-03-17T20:41:12.1488012Z ---------------------------------------------------------------------- 2022-03-17T20:41:12.1488324Z Ran 1 test in 5.717s 2022-03-17T20:41:12.1488485Z 2022-03-17T20:41:12.1488579Z OK 2022-03-17T20:41:12.1488716Z 2022-03-17T20:41:12.1488857Z Generating XML reports... 2022-03-17T20:41:12.1534070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204106.xml 2022-03-17T20:41:13.4410104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:13.4425892Z 2022-03-17T20:41:13.4426194Z Running tests... 2022-03-17T20:41:13.4426643Z ---------------------------------------------------------------------- 2022-03-17T20:41:13.4505346Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip (0.008s) 2022-03-17T20:41:13.4505553Z 2022-03-17T20:41:13.4505883Z ---------------------------------------------------------------------- 2022-03-17T20:41:13.4506196Z Ran 1 test in 0.008s 2022-03-17T20:41:13.4506362Z 2022-03-17T20:41:13.4506476Z OK (skipped=1) 2022-03-17T20:41:13.4506591Z 2022-03-17T20:41:13.4506713Z Generating XML reports... 2022-03-17T20:41:13.4550112Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204113.xml 2022-03-17T20:41:14.5847179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:14.5863422Z 2022-03-17T20:41:14.5863811Z Running tests... 2022-03-17T20:41:14.5864315Z ---------------------------------------------------------------------- 2022-03-17T20:41:16.1211147Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:41:16.1635097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12273 2022-03-17T20:41:16.1762469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12274 2022-03-17T20:41:17.2613847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:17.2618161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:17.2618962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:17.2715081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:17.2721920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:17.3633933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:19.9736243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:41:19.9737002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:41:20.3860040Z ok (5.799s) 2022-03-17T20:41:20.3860403Z 2022-03-17T20:41:20.3861401Z ---------------------------------------------------------------------- 2022-03-17T20:41:20.3861803Z Ran 1 test in 5.800s 2022-03-17T20:41:20.3861926Z 2022-03-17T20:41:20.3862031Z OK 2022-03-17T20:41:20.3862182Z 2022-03-17T20:41:20.3862294Z Generating XML reports... 2022-03-17T20:41:20.3907209Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204114.xml 2022-03-17T20:41:21.6943088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:21.6958669Z 2022-03-17T20:41:21.6959236Z Running tests... 2022-03-17T20:41:21.6959839Z ---------------------------------------------------------------------- 2022-03-17T20:41:23.2279188Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:41:23.2709052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12400 2022-03-17T20:41:23.2838358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12401 2022-03-17T20:41:24.3653426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:24.4022150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:24.4023504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:24.4059224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:24.4066433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:24.5035106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:24.5245982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:41:24.5247016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:41:24.5247961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:41:24.5249223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:41:24.5250289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:41:24.5251370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:41:24.5252055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:41:24.5252737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:41:26.1914734Z ok (4.495s) 2022-03-17T20:41:26.1915159Z 2022-03-17T20:41:26.1915537Z ---------------------------------------------------------------------- 2022-03-17T20:41:26.1915869Z Ran 1 test in 4.496s 2022-03-17T20:41:26.1916027Z 2022-03-17T20:41:26.1916100Z OK 2022-03-17T20:41:26.1916258Z 2022-03-17T20:41:26.1916429Z Generating XML reports... 2022-03-17T20:41:26.1962171Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204121.xml 2022-03-17T20:41:27.4918529Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:27.4933655Z 2022-03-17T20:41:27.4933993Z Running tests... 2022-03-17T20:41:27.4934398Z ---------------------------------------------------------------------- 2022-03-17T20:41:29.0274881Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:41:29.0701553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12530 2022-03-17T20:41:29.0829742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12531 2022-03-17T20:41:30.1221708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:30.1275409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:30.1276218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:30.1323132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:30.1329940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:30.2287067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:30.2497091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:41:30.2497568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:41:30.2498401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:41:30.2499095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:41:30.2501366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:41:30.2602222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:41:30.2602865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:41:30.2603561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:41:41.7349770Z [W ProcessGroupNCCL.cpp:839] [Rank 0] Found key in store: NCCLABORTEDCOMM:20e989ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators 2022-03-17T20:41:41.7350727Z [W ProcessGroupNCCL.cpp:839] [Rank 1] Found key in store: NCCLABORTEDCOMM:20e989ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators 2022-03-17T20:41:42.0082400Z ok (14.515s) 2022-03-17T20:41:42.0082582Z 2022-03-17T20:41:42.0083064Z ---------------------------------------------------------------------- 2022-03-17T20:41:42.0083315Z Ran 1 test in 14.515s 2022-03-17T20:41:42.0083486Z 2022-03-17T20:41:42.0083580Z OK 2022-03-17T20:41:42.0083717Z 2022-03-17T20:41:42.0083827Z Generating XML reports... 2022-03-17T20:41:42.0128427Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204127.xml 2022-03-17T20:41:43.3039174Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:43.3054283Z 2022-03-17T20:41:43.3054432Z Running tests... 2022-03-17T20:41:43.3054908Z ---------------------------------------------------------------------- 2022-03-17T20:41:44.8385435Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:41:44.8814070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12660 2022-03-17T20:41:44.8941947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12661 2022-03-17T20:41:45.9396426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:45.9962704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:45.9963518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:46.0004710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:46.0012485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:46.0977665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:49.5047828Z ok (6.199s) 2022-03-17T20:41:49.5048053Z 2022-03-17T20:41:49.5048439Z ---------------------------------------------------------------------- 2022-03-17T20:41:49.5048758Z Ran 1 test in 6.199s 2022-03-17T20:41:49.5048923Z 2022-03-17T20:41:49.5049016Z OK 2022-03-17T20:41:49.5049151Z 2022-03-17T20:41:49.5049268Z Generating XML reports... 2022-03-17T20:41:49.5093667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204143.xml 2022-03-17T20:41:50.7599831Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:50.7614767Z 2022-03-17T20:41:50.7615062Z Running tests... 2022-03-17T20:41:50.7615513Z ---------------------------------------------------------------------- 2022-03-17T20:41:52.3087292Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:41:52.3514636Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12787 2022-03-17T20:41:52.3642344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12788 2022-03-17T20:41:53.4404692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:41:53.4408849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:41:53.4409605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:53.4505855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:41:53.4512518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:41:53.5423317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:41:56.8750295Z ok (6.113s) 2022-03-17T20:41:56.8750792Z 2022-03-17T20:41:56.8751454Z ---------------------------------------------------------------------- 2022-03-17T20:41:56.8752030Z Ran 1 test in 6.113s 2022-03-17T20:41:56.8752304Z 2022-03-17T20:41:56.8752483Z OK 2022-03-17T20:41:56.8753128Z 2022-03-17T20:41:56.8753490Z Generating XML reports... 2022-03-17T20:41:56.8800419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204150.xml 2022-03-17T20:41:58.1794566Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:41:58.1811262Z 2022-03-17T20:41:58.1811687Z Running tests... 2022-03-17T20:41:58.1812192Z ---------------------------------------------------------------------- 2022-03-17T20:41:59.7197718Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:41:59.7621722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12914 2022-03-17T20:41:59.7748035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12915 2022-03-17T20:42:00.8216194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:00.8428491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:00.8429307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:00.8521504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:00.8527832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:00.9443575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:03.9849198Z ok (5.803s) 2022-03-17T20:42:03.9849427Z 2022-03-17T20:42:03.9849842Z ---------------------------------------------------------------------- 2022-03-17T20:42:03.9850187Z Ran 1 test in 5.804s 2022-03-17T20:42:03.9850339Z 2022-03-17T20:42:03.9850422Z OK 2022-03-17T20:42:03.9850547Z 2022-03-17T20:42:03.9850682Z Generating XML reports... 2022-03-17T20:42:03.9895948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204158.xml 2022-03-17T20:42:05.2938892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:05.2954291Z 2022-03-17T20:42:05.2954508Z Running tests... 2022-03-17T20:42:05.2954945Z ---------------------------------------------------------------------- 2022-03-17T20:42:06.8243450Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:06.8664356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13037 2022-03-17T20:42:06.8792203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13038 2022-03-17T20:42:07.9435874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:07.9465040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:07.9466172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:07.9537364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:07.9544858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:08.0480991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:10.6561205Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:42:10.6714438Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:42:10.6776835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:10.6777322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:10.9888194Z ok (5.693s) 2022-03-17T20:42:10.9888380Z 2022-03-17T20:42:10.9888750Z ---------------------------------------------------------------------- 2022-03-17T20:42:10.9889117Z Ran 1 test in 5.693s 2022-03-17T20:42:10.9889286Z 2022-03-17T20:42:10.9889372Z OK 2022-03-17T20:42:10.9889501Z 2022-03-17T20:42:10.9889626Z Generating XML reports... 2022-03-17T20:42:10.9934629Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204205.xml 2022-03-17T20:42:12.2948415Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:12.2968102Z 2022-03-17T20:42:12.2968732Z Running tests... 2022-03-17T20:42:12.2969215Z ---------------------------------------------------------------------- 2022-03-17T20:42:13.8308416Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:13.8752638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13164 2022-03-17T20:42:13.8882361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13165 2022-03-17T20:42:14.9603372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:14.9966385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:14.9967171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:15.0007844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:15.0014583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:15.0981824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:17.4187193Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:42:17.4188093Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:42:17.4189243Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:42:17.4190065Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:42:18.0982041Z ok (5.801s) 2022-03-17T20:42:18.0982405Z 2022-03-17T20:42:18.0983010Z ---------------------------------------------------------------------- 2022-03-17T20:42:18.0983621Z Ran 1 test in 5.801s 2022-03-17T20:42:18.0983936Z 2022-03-17T20:42:18.0984098Z OK 2022-03-17T20:42:18.0984354Z 2022-03-17T20:42:18.0984588Z Generating XML reports... 2022-03-17T20:42:18.1031398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204212.xml 2022-03-17T20:42:19.3864849Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:19.3879591Z 2022-03-17T20:42:19.3880039Z Running tests... 2022-03-17T20:42:19.3880544Z ---------------------------------------------------------------------- 2022-03-17T20:42:20.9010685Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:20.9437128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13291 2022-03-17T20:42:20.9563693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13292 2022-03-17T20:42:22.0240638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:22.0516703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:22.0517538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:22.0543912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:22.0550599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:22.1532080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:24.1734589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:24.1735112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:24.2040507Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:42:24.2042085Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:42:24.5650500Z ok (5.177s) 2022-03-17T20:42:24.5650733Z 2022-03-17T20:42:24.5651119Z ---------------------------------------------------------------------- 2022-03-17T20:42:24.5651514Z Ran 1 test in 5.177s 2022-03-17T20:42:24.5651608Z 2022-03-17T20:42:24.5651703Z OK 2022-03-17T20:42:24.5651819Z 2022-03-17T20:42:24.5651959Z Generating XML reports... 2022-03-17T20:42:24.5697268Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204219.xml 2022-03-17T20:42:25.8471159Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:25.8485903Z 2022-03-17T20:42:25.8486341Z Running tests... 2022-03-17T20:42:25.8486846Z ---------------------------------------------------------------------- 2022-03-17T20:42:27.3720671Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:27.4142188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13421 2022-03-17T20:42:27.4269272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13422 2022-03-17T20:42:28.4678530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:28.5098327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:28.5099366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:28.5186209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:28.5193012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:28.6113921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:30.6542975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:30.6543512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:30.6873481Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:42:30.6875487Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:42:31.0358164Z ok (5.187s) 2022-03-17T20:42:31.0358426Z 2022-03-17T20:42:31.0358795Z ---------------------------------------------------------------------- 2022-03-17T20:42:31.0359106Z Ran 1 test in 5.187s 2022-03-17T20:42:31.0359328Z 2022-03-17T20:42:31.0359352Z OK 2022-03-17T20:42:31.0359477Z 2022-03-17T20:42:31.0359593Z Generating XML reports... 2022-03-17T20:42:31.0404364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204225.xml 2022-03-17T20:42:32.3366524Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:32.3381997Z 2022-03-17T20:42:32.3382224Z Running tests... 2022-03-17T20:42:32.3382675Z ---------------------------------------------------------------------- 2022-03-17T20:42:33.8618736Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:33.9042363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13551 2022-03-17T20:42:33.9167550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13552 2022-03-17T20:42:34.9425693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:35.0047920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:35.0048738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:35.0135158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:35.0141686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:35.1062950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:37.7257892Z ok (5.387s) 2022-03-17T20:42:37.7258097Z 2022-03-17T20:42:37.7258734Z ---------------------------------------------------------------------- 2022-03-17T20:42:37.7259064Z Ran 1 test in 5.388s 2022-03-17T20:42:37.7259357Z 2022-03-17T20:42:37.7259516Z OK 2022-03-17T20:42:37.7259744Z 2022-03-17T20:42:37.7259969Z Generating XML reports... 2022-03-17T20:42:37.7304950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204232.xml 2022-03-17T20:42:39.0296191Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:39.0311802Z 2022-03-17T20:42:39.0312480Z Running tests... 2022-03-17T20:42:39.0313392Z ---------------------------------------------------------------------- 2022-03-17T20:42:40.5783379Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:40.6211789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13674 2022-03-17T20:42:40.6339097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13675 2022-03-17T20:42:41.6966333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:41.7103057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:41.7104601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:41.7169126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:41.7175565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:41.8119124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:43.1382340Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:42:43.1383281Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:42:43.1384436Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:42:43.1385243Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:42:43.4287260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:43.4288104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:43.4339803Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:42:43.4443006Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:42:43.4444036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:43.4444508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:43.4502142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:43.4503104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:43.7419776Z ok (4.710s) 2022-03-17T20:42:43.7420127Z 2022-03-17T20:42:43.7421145Z ---------------------------------------------------------------------- 2022-03-17T20:42:43.7421618Z Ran 1 test in 4.711s 2022-03-17T20:42:43.7421775Z 2022-03-17T20:42:43.7421875Z OK 2022-03-17T20:42:43.7422006Z 2022-03-17T20:42:43.7422140Z Generating XML reports... 2022-03-17T20:42:43.7466836Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204239.xml 2022-03-17T20:42:45.0308087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:45.0322489Z 2022-03-17T20:42:45.0322659Z Running tests... 2022-03-17T20:42:45.0323519Z ---------------------------------------------------------------------- 2022-03-17T20:42:46.5447705Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:46.5867504Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13800 2022-03-17T20:42:46.5993660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13801 2022-03-17T20:42:47.6516287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:47.6531216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:47.6532042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:47.6616930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:47.6623775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:47.7546731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:49.0881516Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:42:49.0882419Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:42:49.0883570Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T20:42:49.0884386Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T20:42:49.3767378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:49.3767908Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:49.7074115Z ok (4.675s) 2022-03-17T20:42:49.7074318Z 2022-03-17T20:42:49.7074733Z ---------------------------------------------------------------------- 2022-03-17T20:42:49.7075123Z Ran 1 test in 4.675s 2022-03-17T20:42:49.7075280Z 2022-03-17T20:42:49.7075404Z OK 2022-03-17T20:42:49.7075520Z 2022-03-17T20:42:49.7075632Z Generating XML reports... 2022-03-17T20:42:49.7124056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204245.xml 2022-03-17T20:42:51.0129323Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:51.0145322Z 2022-03-17T20:42:51.0145590Z Running tests... 2022-03-17T20:42:51.0146049Z ---------------------------------------------------------------------- 2022-03-17T20:42:52.5420534Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:52.5843649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13926 2022-03-17T20:42:52.5969286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13927 2022-03-17T20:42:53.6487397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:53.6488460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:53.6489502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:53.6590571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:53.6597854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:42:53.7503811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:55.3513118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:55.3514577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:55.3592803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:55.3596203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:42:55.7046684Z ok (4.690s) 2022-03-17T20:42:55.7046877Z 2022-03-17T20:42:55.7047261Z ---------------------------------------------------------------------- 2022-03-17T20:42:55.7047652Z Ran 1 test in 4.690s 2022-03-17T20:42:55.7047819Z 2022-03-17T20:42:55.7047895Z OK 2022-03-17T20:42:55.7048033Z 2022-03-17T20:42:55.7048158Z Generating XML reports... 2022-03-17T20:42:55.7093423Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204251.xml 2022-03-17T20:42:57.0084394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:42:57.0100095Z 2022-03-17T20:42:57.0100382Z Running tests... 2022-03-17T20:42:57.0101124Z ---------------------------------------------------------------------- 2022-03-17T20:42:58.5397979Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:42:58.5818612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14052 2022-03-17T20:42:58.5946569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14053 2022-03-17T20:42:59.6641679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:42:59.7048151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:42:59.7048939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:59.7150856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:42:59.7157215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:42:59.8063790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:01.4907243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:01.4907790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:02.4041362Z ok (5.394s) 2022-03-17T20:43:02.4041593Z 2022-03-17T20:43:02.4042256Z ---------------------------------------------------------------------- 2022-03-17T20:43:02.4042504Z Ran 1 test in 5.394s 2022-03-17T20:43:02.4042650Z 2022-03-17T20:43:02.4042740Z OK 2022-03-17T20:43:02.4042874Z 2022-03-17T20:43:02.4042986Z Generating XML reports... 2022-03-17T20:43:02.4086852Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204257.xml 2022-03-17T20:43:03.7034261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:03.7049100Z 2022-03-17T20:43:03.7049251Z Running tests... 2022-03-17T20:43:03.7050139Z ---------------------------------------------------------------------- 2022-03-17T20:43:05.2556262Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:43:05.2982855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14182 2022-03-17T20:43:05.3110543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14183 2022-03-17T20:43:06.3944989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:43:06.4179045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:43:06.4179994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:06.4248938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:06.4255530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:43:06.5194052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:09.1203698Z ok (5.415s) 2022-03-17T20:43:09.1203895Z 2022-03-17T20:43:09.1204291Z ---------------------------------------------------------------------- 2022-03-17T20:43:09.1204630Z Ran 1 test in 5.415s 2022-03-17T20:43:09.1204783Z 2022-03-17T20:43:09.1204870Z OK 2022-03-17T20:43:09.1207440Z 2022-03-17T20:43:09.1207845Z Generating XML reports... 2022-03-17T20:43:09.1248953Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204303.xml 2022-03-17T20:43:10.4255609Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:10.4271259Z 2022-03-17T20:43:10.4271379Z Running tests... 2022-03-17T20:43:10.4272063Z ---------------------------------------------------------------------- 2022-03-17T20:43:11.9672712Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:43:12.0094907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14305 2022-03-17T20:43:12.0220773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14306 2022-03-17T20:43:13.0858297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:43:13.1218284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:43:13.1219090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:13.1265547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:13.1272104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:43:13.2232958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:15.8312395Z ok (5.404s) 2022-03-17T20:43:15.8312667Z 2022-03-17T20:43:15.8313258Z ---------------------------------------------------------------------- 2022-03-17T20:43:15.8313512Z Ran 1 test in 5.404s 2022-03-17T20:43:15.8313682Z 2022-03-17T20:43:15.8313774Z OK 2022-03-17T20:43:15.8313890Z 2022-03-17T20:43:15.8314049Z Generating XML reports... 2022-03-17T20:43:15.8357827Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204310.xml 2022-03-17T20:43:17.1298183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:17.1313431Z 2022-03-17T20:43:17.1313853Z Running tests... 2022-03-17T20:43:17.1314354Z ---------------------------------------------------------------------- 2022-03-17T20:43:18.6638142Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:43:18.7062011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14428 2022-03-17T20:43:18.7190154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14429 2022-03-17T20:43:19.8113973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:43:19.8114456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:43:19.8115249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:19.8116207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:19.8122505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:19.8122997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:43:22.4439691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:22.4440265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:22.8286912Z ok (5.697s) 2022-03-17T20:43:22.8287147Z 2022-03-17T20:43:22.8287575Z ---------------------------------------------------------------------- 2022-03-17T20:43:22.8287922Z Ran 1 test in 5.697s 2022-03-17T20:43:22.8288089Z 2022-03-17T20:43:22.8288195Z OK 2022-03-17T20:43:22.8288297Z 2022-03-17T20:43:22.8288396Z Generating XML reports... 2022-03-17T20:43:22.8334205Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204317.xml 2022-03-17T20:43:24.1360537Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:43:24.1376140Z 2022-03-17T20:43:24.1376521Z Running tests... 2022-03-17T20:43:24.1377028Z ---------------------------------------------------------------------- 2022-03-17T20:43:25.6643603Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:43:25.7071443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14555 2022-03-17T20:43:25.7198604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14556 2022-03-17T20:43:26.7965923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:43:26.7966744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:26.7967323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:43:26.7967980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:43:26.7974180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:43:26.7974675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:43:28.1109009Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:43:28.1112067Z Running test: batch_norm_net sync interval 2022-03-17T20:43:28.1112403Z 1 with iteration mapping 2022-03-17T20:43:28.1112815Z {0: 0, 1: 2} 2022-03-17T20:43:28.1406362Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T20:43:28.8351729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:28.8385668Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:28.8386425Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:28.8387723Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:28.8388526Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:28.8466452Z Running test: batch_norm_net sync interval 2022-03-17T20:43:28.8466789Z 1 with iteration mapping 2022-03-17T20:43:28.8467058Z {0: 0, 1: 3} 2022-03-17T20:43:28.8507178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:28.8555198Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:28.8555946Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:28.8556972Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:28.8557720Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:28.8622017Z Running test: batch_norm_net sync interval 2022-03-17T20:43:28.8622366Z 1 with iteration mapping 2022-03-17T20:43:28.8622614Z {0: 0, 1: 10} 2022-03-17T20:43:28.8661512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:28.8850318Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:28.8851036Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:28.8852064Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:28.8852824Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:28.8917636Z Running test: batch_norm_net sync interval 2022-03-17T20:43:28.8918033Z 1 with iteration mapping 2022-03-17T20:43:28.8918311Z {0: 5, 1: 7} 2022-03-17T20:43:29.5217254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:29.5217758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:29.5347167Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:29.5347911Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:29.5349110Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:29.5349889Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:29.5416158Z Running test: batch_norm_net sync interval 2022-03-17T20:43:29.5416488Z 1 with iteration mapping 2022-03-17T20:43:29.5416727Z {0: 5, 1: 8} 2022-03-17T20:43:29.5456032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:29.5456518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:29.5604108Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:29.5604876Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:29.5605901Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:29.5606662Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:29.5671248Z Running test: batch_norm_net sync interval 2022-03-17T20:43:29.5671557Z 1 with iteration mapping 2022-03-17T20:43:29.5671827Z {0: 5, 1: 15} 2022-03-17T20:43:29.5710711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:29.5711209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:29.5993209Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:29.5993979Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:29.5994981Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:29.5995757Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:29.6059705Z Running test: large_conv_model sync interval 2022-03-17T20:43:29.6059994Z 1 with iteration mapping 2022-03-17T20:43:29.6060518Z {0: 0, 1: 2} 2022-03-17T20:43:31.1060519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:32.1920374Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:32.1921154Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:32.1922323Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:32.1923121Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:32.1990865Z Running test: large_conv_model sync interval 2022-03-17T20:43:32.1991161Z 1 with iteration mapping 2022-03-17T20:43:32.1991440Z {0: 0, 1: 3} 2022-03-17T20:43:33.2719969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:35.4493919Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:35.4494693Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:35.4495693Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:35.4496485Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:35.4558597Z Running test: large_conv_model sync interval 2022-03-17T20:43:35.4558909Z 1 with iteration mapping 2022-03-17T20:43:35.4559169Z {0: 0, 1: 10} 2022-03-17T20:43:36.5236896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:46.2984142Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:46.2984987Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:46.2986041Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:46.2986788Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:46.3049544Z Running test: large_conv_model sync interval 2022-03-17T20:43:46.3049869Z 1 with iteration mapping 2022-03-17T20:43:46.3050129Z {0: 5, 1: 7} 2022-03-17T20:43:47.7998835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:47.7999363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:54.3527197Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:54.3528268Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:54.3529399Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:43:54.3530169Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:43:54.3592767Z Running test: large_conv_model sync interval 2022-03-17T20:43:54.3593093Z 1 with iteration mapping 2022-03-17T20:43:54.3593355Z {0: 5, 1: 8} 2022-03-17T20:43:55.4628334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:43:55.4628830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:03.0971160Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:03.0972020Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:03.0973041Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:03.0973813Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:03.1036591Z Running test: large_conv_model sync interval 2022-03-17T20:44:03.1036897Z 1 with iteration mapping 2022-03-17T20:44:03.1037176Z {0: 5, 1: 15} 2022-03-17T20:44:04.1986609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:04.1987115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.4434551Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.4435354Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.4436395Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.4437156Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.4499828Z Running test: small_model sync interval 2022-03-17T20:44:19.4500144Z 1 with iteration mapping 2022-03-17T20:44:19.4500695Z {0: 0, 1: 2} 2022-03-17T20:44:19.4549586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.4633601Z Running test: small_model sync interval 2022-03-17T20:44:19.4634131Z 1 with iteration mapping 2022-03-17T20:44:19.4634381Z {0: 0, 1: 3} 2022-03-17T20:44:19.4670933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.4773754Z Running test: small_model sync interval 2022-03-17T20:44:19.4774075Z 1 with iteration mapping 2022-03-17T20:44:19.4774346Z {0: 0, 1: 10} 2022-03-17T20:44:19.4810937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.5042055Z Running test: small_model sync interval 2022-03-17T20:44:19.5042485Z 1 with iteration mapping 2022-03-17T20:44:19.5042777Z {0: 5, 1: 7} 2022-03-17T20:44:19.5080547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.5081015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.5262239Z Running test: small_model sync interval 2022-03-17T20:44:19.5262566Z 1 with iteration mapping 2022-03-17T20:44:19.5262815Z {0: 5, 1: 8} 2022-03-17T20:44:19.5300469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.5300940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.5501200Z Running test: small_model sync interval 2022-03-17T20:44:19.5501517Z 1 with iteration mapping 2022-03-17T20:44:19.5501766Z {0: 5, 1: 15} 2022-03-17T20:44:19.5539261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.5539726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.5865534Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.5865899Z 1 with iteration mapping 2022-03-17T20:44:19.5866150Z {0: 0, 1: 2} 2022-03-17T20:44:19.5914563Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.5915350Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.5916354Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.5917127Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.5936561Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.5936900Z 1 with iteration mapping 2022-03-17T20:44:19.5937170Z {0: 0, 1: 3} 2022-03-17T20:44:19.5994165Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.5994937Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.5995951Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.5996858Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6016523Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.6016862Z 1 with iteration mapping 2022-03-17T20:44:19.6017133Z {0: 0, 1: 10} 2022-03-17T20:44:19.6155385Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6156190Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6157208Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6157954Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6178439Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.6178797Z 1 with iteration mapping 2022-03-17T20:44:19.6179050Z {0: 5, 1: 7} 2022-03-17T20:44:19.6197583Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:44:19.6286626Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6287416Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6288418Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6289187Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6309684Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.6310021Z 1 with iteration mapping 2022-03-17T20:44:19.6310292Z {0: 5, 1: 8} 2022-03-17T20:44:19.6425540Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6426324Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6427317Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6428198Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6448358Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.6448701Z 1 with iteration mapping 2022-03-17T20:44:19.6448971Z {0: 5, 1: 15} 2022-03-17T20:44:19.6642790Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6643592Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6644608Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6645358Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6665152Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.6665539Z 1 with iteration mapping 2022-03-17T20:44:19.6665793Z {0: 0, 1: 2} 2022-03-17T20:44:19.6684696Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:44:19.6713306Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6714093Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6715108Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6715859Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6735519Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.6735871Z 1 with iteration mapping 2022-03-17T20:44:19.6736124Z {0: 0, 1: 3} 2022-03-17T20:44:19.6794014Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6794795Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6795792Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6796682Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6816197Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.6816530Z 1 with iteration mapping 2022-03-17T20:44:19.6816806Z {0: 0, 1: 10} 2022-03-17T20:44:19.6959428Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6960201Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6961228Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.6961986Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.6981538Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.6981897Z 1 with iteration mapping 2022-03-17T20:44:19.6982165Z {0: 5, 1: 7} 2022-03-17T20:44:19.7088365Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.7089148Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.7090163Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.7090902Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.7111499Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.7111845Z 1 with iteration mapping 2022-03-17T20:44:19.7112093Z {0: 5, 1: 8} 2022-03-17T20:44:19.7231288Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.7232064Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.7233063Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.7233819Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.7253500Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:44:19.7253948Z 1 with iteration mapping 2022-03-17T20:44:19.7254222Z {0: 5, 1: 15} 2022-03-17T20:44:19.7454007Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.7454771Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.7455884Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:19.7456661Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:19.7477176Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:44:19.7477517Z 1 with iteration mapping 2022-03-17T20:44:19.7477781Z {0: 0, 1: 2} 2022-03-17T20:44:19.7487812Z registered hook 2022-03-17T20:44:19.7488182Z registered hook 2022-03-17T20:44:19.7514994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.7600374Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:44:19.7600712Z 1 with iteration mapping 2022-03-17T20:44:19.7600966Z {0: 0, 1: 3} 2022-03-17T20:44:19.7610879Z registered hook 2022-03-17T20:44:19.7611239Z registered hook 2022-03-17T20:44:19.7638502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.7742730Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:44:19.7743045Z 1 with iteration mapping 2022-03-17T20:44:19.7743311Z {0: 0, 1: 10} 2022-03-17T20:44:19.7753377Z registered hook 2022-03-17T20:44:19.7753737Z registered hook 2022-03-17T20:44:19.7780828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.8018024Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:44:19.8018369Z 1 with iteration mapping 2022-03-17T20:44:19.8018638Z {0: 5, 1: 7} 2022-03-17T20:44:19.8028682Z registered hook 2022-03-17T20:44:19.8029055Z registered hook 2022-03-17T20:44:19.8057309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.8057781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.8243991Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:44:19.8244306Z 1 with iteration mapping 2022-03-17T20:44:19.8244570Z {0: 5, 1: 8} 2022-03-17T20:44:19.8254208Z registered hook 2022-03-17T20:44:19.8254559Z registered hook 2022-03-17T20:44:19.8283046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.8283515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.8489398Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:44:19.8489727Z 1 with iteration mapping 2022-03-17T20:44:19.8490117Z {0: 5, 1: 15} 2022-03-17T20:44:19.8499436Z registered hook 2022-03-17T20:44:19.8499808Z registered hook 2022-03-17T20:44:19.8528412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.8528896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.8885462Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:44:19.8885807Z 1 with iteration mapping 2022-03-17T20:44:19.8886073Z {0: 0, 1: 2} 2022-03-17T20:44:19.8895892Z registered hook 2022-03-17T20:44:19.8896279Z registered hook 2022-03-17T20:44:19.8904046Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T20:44:19.8907956Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T20:44:19.8925234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.8928727Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T20:44:19.8931992Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T20:44:19.9018642Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:44:19.9018988Z 1 with iteration mapping 2022-03-17T20:44:19.9019241Z {0: 0, 1: 3} 2022-03-17T20:44:19.9029405Z registered hook 2022-03-17T20:44:19.9029837Z registered hook 2022-03-17T20:44:19.9060902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.9172519Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:44:19.9172853Z 1 with iteration mapping 2022-03-17T20:44:19.9173112Z {0: 0, 1: 10} 2022-03-17T20:44:19.9183089Z registered hook 2022-03-17T20:44:19.9183769Z registered hook 2022-03-17T20:44:19.9214043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.9484835Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:44:19.9485194Z 1 with iteration mapping 2022-03-17T20:44:19.9485463Z {0: 5, 1: 7} 2022-03-17T20:44:19.9495152Z registered hook 2022-03-17T20:44:19.9495582Z registered hook 2022-03-17T20:44:19.9529926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.9530404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.9744236Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:44:19.9744570Z 1 with iteration mapping 2022-03-17T20:44:19.9744821Z {0: 5, 1: 8} 2022-03-17T20:44:19.9754550Z registered hook 2022-03-17T20:44:19.9754947Z registered hook 2022-03-17T20:44:19.9788906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:19.9789438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.0025945Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:44:20.0026522Z 1 with iteration mapping 2022-03-17T20:44:20.0026792Z {0: 5, 1: 15} 2022-03-17T20:44:20.0036373Z registered hook 2022-03-17T20:44:20.0036751Z registered hook 2022-03-17T20:44:20.0070209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.0070718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.0426604Z Running test: batch_norm_net sync interval 2022-03-17T20:44:20.0426912Z 2 with iteration mapping 2022-03-17T20:44:20.0427296Z {0: 0, 1: 2} 2022-03-17T20:44:20.0466792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.0493195Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.0493934Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.0494958Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.0495713Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.0572852Z Running test: batch_norm_net sync interval 2022-03-17T20:44:20.0573200Z 2 with iteration mapping 2022-03-17T20:44:20.0573478Z {0: 0, 1: 3} 2022-03-17T20:44:20.0601576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.0642683Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.0643388Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.0644429Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.0645192Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.0710749Z Running test: batch_norm_net sync interval 2022-03-17T20:44:20.0711135Z 2 with iteration mapping 2022-03-17T20:44:20.0711418Z {0: 0, 1: 10} 2022-03-17T20:44:20.0749859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.0904744Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.0905474Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.0906479Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.0907415Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.0972491Z Running test: batch_norm_net sync interval 2022-03-17T20:44:20.0972829Z 2 with iteration mapping 2022-03-17T20:44:20.0973100Z {0: 5, 1: 7} 2022-03-17T20:44:20.1012768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.1013394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.1118807Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.1119527Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.1120536Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.1121305Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.1188091Z Running test: batch_norm_net sync interval 2022-03-17T20:44:20.1188400Z 2 with iteration mapping 2022-03-17T20:44:20.1188681Z {0: 5, 1: 8} 2022-03-17T20:44:20.1227436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.1227946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.1350651Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.1351361Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.1352372Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.1353130Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.1420878Z Running test: batch_norm_net sync interval 2022-03-17T20:44:20.1421215Z 2 with iteration mapping 2022-03-17T20:44:20.1421494Z {0: 5, 1: 15} 2022-03-17T20:44:20.1460316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.1460818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:20.1695318Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.1696034Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.1697198Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:20.1697959Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:20.1763663Z Running test: large_conv_model sync interval 2022-03-17T20:44:20.1763987Z 3 with iteration mapping 2022-03-17T20:44:20.1764272Z {0: 0, 1: 2} 2022-03-17T20:44:21.2560802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:22.3534054Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:22.3534837Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:22.3535892Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:22.3536634Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:22.3607888Z Running test: large_conv_model sync interval 2022-03-17T20:44:22.3608246Z 3 with iteration mapping 2022-03-17T20:44:22.3608524Z {0: 0, 1: 3} 2022-03-17T20:44:23.4428546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:25.6163073Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:25.6163807Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:25.6164858Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:25.6165606Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:25.6229237Z Running test: large_conv_model sync interval 2022-03-17T20:44:25.6229625Z 3 with iteration mapping 2022-03-17T20:44:25.6229882Z {0: 0, 1: 10} 2022-03-17T20:44:26.6920666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:36.4702876Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:36.4703717Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:36.4704768Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:36.4705802Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:36.4768657Z Running test: large_conv_model sync interval 2022-03-17T20:44:36.4768984Z 3 with iteration mapping 2022-03-17T20:44:36.4769242Z {0: 5, 1: 7} 2022-03-17T20:44:37.5695213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:37.5695733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:44.1297833Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:44.1298705Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:44.1299743Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:44.1300818Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:44.1363878Z Running test: large_conv_model sync interval 2022-03-17T20:44:44.1364209Z 3 with iteration mapping 2022-03-17T20:44:44.1364469Z {0: 5, 1: 8} 2022-03-17T20:44:45.2253368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:45.2253865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:52.8518720Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:52.8519549Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:52.8520587Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:44:52.8521361Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:44:52.8585501Z Running test: large_conv_model sync interval 2022-03-17T20:44:52.8585806Z 3 with iteration mapping 2022-03-17T20:44:52.8586087Z {0: 5, 1: 15} 2022-03-17T20:44:53.9457880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:44:53.9458389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.1793869Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.1794686Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.1795726Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.1796774Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.1859645Z Running test: small_model sync interval 2022-03-17T20:45:09.1859974Z 4 with iteration mapping 2022-03-17T20:45:09.1860251Z {0: 0, 1: 2} 2022-03-17T20:45:09.1898495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.1973096Z Running test: small_model sync interval 2022-03-17T20:45:09.1973405Z 4 with iteration mapping 2022-03-17T20:45:09.1973682Z {0: 0, 1: 3} 2022-03-17T20:45:09.2009895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.2092385Z Running test: small_model sync interval 2022-03-17T20:45:09.2092691Z 4 with iteration mapping 2022-03-17T20:45:09.2092962Z {0: 0, 1: 10} 2022-03-17T20:45:09.2129172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.2289340Z Running test: small_model sync interval 2022-03-17T20:45:09.2289660Z 4 with iteration mapping 2022-03-17T20:45:09.2289914Z {0: 5, 1: 7} 2022-03-17T20:45:09.2327552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.2328032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.2455253Z Running test: small_model sync interval 2022-03-17T20:45:09.2455569Z 4 with iteration mapping 2022-03-17T20:45:09.2455825Z {0: 5, 1: 8} 2022-03-17T20:45:09.2493314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.2493798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.2628205Z Running test: small_model sync interval 2022-03-17T20:45:09.2628521Z 4 with iteration mapping 2022-03-17T20:45:09.2628777Z {0: 5, 1: 15} 2022-03-17T20:45:09.2665923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.2666412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.2877699Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.2878049Z 5 with iteration mapping 2022-03-17T20:45:09.2878306Z {0: 0, 1: 2} 2022-03-17T20:45:09.2921072Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.2921875Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.2922902Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.2923656Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.2943036Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.2943388Z 5 with iteration mapping 2022-03-17T20:45:09.2943799Z {0: 0, 1: 3} 2022-03-17T20:45:09.2993679Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.2994466Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.2995579Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.2996353Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3015813Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3016146Z 5 with iteration mapping 2022-03-17T20:45:09.3016421Z {0: 0, 1: 10} 2022-03-17T20:45:09.3129281Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3130057Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3131074Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3131838Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3149595Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3149947Z 5 with iteration mapping 2022-03-17T20:45:09.3150224Z {0: 5, 1: 7} 2022-03-17T20:45:09.3237301Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3238080Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3239095Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3239845Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3259810Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3260161Z 5 with iteration mapping 2022-03-17T20:45:09.3260657Z {0: 5, 1: 8} 2022-03-17T20:45:09.3354247Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3355027Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3356166Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3356925Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3376562Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3376928Z 5 with iteration mapping 2022-03-17T20:45:09.3377310Z {0: 5, 1: 15} 2022-03-17T20:45:09.3531052Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3531818Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3532842Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3533596Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3553685Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3554037Z 6 with iteration mapping 2022-03-17T20:45:09.3554306Z {0: 0, 1: 2} 2022-03-17T20:45:09.3597178Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3597955Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3598978Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3599722Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3619312Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3619657Z 6 with iteration mapping 2022-03-17T20:45:09.3619919Z {0: 0, 1: 3} 2022-03-17T20:45:09.3671783Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3672557Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3673570Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3674337Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3694047Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3694377Z 6 with iteration mapping 2022-03-17T20:45:09.3694657Z {0: 0, 1: 10} 2022-03-17T20:45:09.3811196Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3812062Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3813107Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3813868Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3833878Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3834236Z 6 with iteration mapping 2022-03-17T20:45:09.3834514Z {0: 5, 1: 7} 2022-03-17T20:45:09.3924564Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3925352Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3926360Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.3927115Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.3947246Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.3947596Z 6 with iteration mapping 2022-03-17T20:45:09.3947853Z {0: 5, 1: 8} 2022-03-17T20:45:09.4047316Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.4048100Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.4049128Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.4049867Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.4070089Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:09.4070437Z 6 with iteration mapping 2022-03-17T20:45:09.4070701Z {0: 5, 1: 15} 2022-03-17T20:45:09.4234411Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.4235328Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.4236318Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:09.4237072Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:09.4256769Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:09.4257096Z 7 with iteration mapping 2022-03-17T20:45:09.4257374Z {0: 0, 1: 2} 2022-03-17T20:45:09.4293919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.4368500Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:09.4368836Z 7 with iteration mapping 2022-03-17T20:45:09.4369089Z {0: 0, 1: 3} 2022-03-17T20:45:09.4405423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.4488211Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:09.4488527Z 7 with iteration mapping 2022-03-17T20:45:09.4488800Z {0: 0, 1: 10} 2022-03-17T20:45:09.4525074Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.4674320Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:09.4674658Z 7 with iteration mapping 2022-03-17T20:45:09.4674907Z {0: 5, 1: 7} 2022-03-17T20:45:09.4711850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.4712327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.4827600Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:09.4827932Z 7 with iteration mapping 2022-03-17T20:45:09.4828184Z {0: 5, 1: 8} 2022-03-17T20:45:09.4865330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.4865822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.4999482Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:09.4999822Z 7 with iteration mapping 2022-03-17T20:45:09.5000072Z {0: 5, 1: 15} 2022-03-17T20:45:09.5036869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5037358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5240305Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:09.5240638Z 8 with iteration mapping 2022-03-17T20:45:09.5240888Z {0: 0, 1: 2} 2022-03-17T20:45:09.5277250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5351643Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:09.5351955Z 8 with iteration mapping 2022-03-17T20:45:09.5352230Z {0: 0, 1: 3} 2022-03-17T20:45:09.5388519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5471116Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:09.5471449Z 8 with iteration mapping 2022-03-17T20:45:09.5471699Z {0: 0, 1: 10} 2022-03-17T20:45:09.5508090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5658100Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:09.5658417Z 8 with iteration mapping 2022-03-17T20:45:09.5658696Z {0: 5, 1: 7} 2022-03-17T20:45:09.5695860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5696360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5811987Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:09.5812304Z 8 with iteration mapping 2022-03-17T20:45:09.5812686Z {0: 5, 1: 8} 2022-03-17T20:45:09.5849429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5849897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.5973215Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:09.5973531Z 8 with iteration mapping 2022-03-17T20:45:09.5973809Z {0: 5, 1: 15} 2022-03-17T20:45:09.6010362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.6010834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.6200807Z Running test: batch_norm_net sync interval 2022-03-17T20:45:09.6201133Z 1 with iteration mapping 2022-03-17T20:45:09.6201381Z {0: 0, 1: 2} 2022-03-17T20:45:09.6222726Z Running test: batch_norm_net sync interval 2022-03-17T20:45:09.6223030Z 1 with iteration mapping 2022-03-17T20:45:09.6223296Z {0: 0, 1: 3} 2022-03-17T20:45:09.6243308Z Running test: batch_norm_net sync interval 2022-03-17T20:45:09.6243597Z 1 with iteration mapping 2022-03-17T20:45:09.6243875Z {0: 0, 1: 10} 2022-03-17T20:45:09.6263293Z Running test: batch_norm_net sync interval 2022-03-17T20:45:09.6263615Z 1 with iteration mapping 2022-03-17T20:45:09.6263885Z {0: 5, 1: 7} 2022-03-17T20:45:09.6306672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.6307161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.6394414Z Running test: batch_norm_net sync interval 2022-03-17T20:45:09.6394711Z 1 with iteration mapping 2022-03-17T20:45:09.6394995Z {0: 5, 1: 8} 2022-03-17T20:45:09.6436832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.6437300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.6524388Z Running test: batch_norm_net sync interval 2022-03-17T20:45:09.6524700Z 1 with iteration mapping 2022-03-17T20:45:09.6524973Z {0: 5, 1: 15} 2022-03-17T20:45:09.6566851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.6567320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:09.6653643Z Running test: large_conv_model sync interval 2022-03-17T20:45:09.6653943Z 1 with iteration mapping 2022-03-17T20:45:09.6654219Z {0: 0, 1: 2} 2022-03-17T20:45:09.6674741Z Running test: large_conv_model sync interval 2022-03-17T20:45:09.6675046Z 1 with iteration mapping 2022-03-17T20:45:09.6675317Z {0: 0, 1: 3} 2022-03-17T20:45:09.6695522Z Running test: large_conv_model sync interval 2022-03-17T20:45:09.6695821Z 1 with iteration mapping 2022-03-17T20:45:09.6696091Z {0: 0, 1: 10} 2022-03-17T20:45:09.6716439Z Running test: large_conv_model sync interval 2022-03-17T20:45:09.6716765Z 1 with iteration mapping 2022-03-17T20:45:09.6717042Z {0: 5, 1: 7} 2022-03-17T20:45:10.7735231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:10.7735741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:15.1663752Z Running test: large_conv_model sync interval 2022-03-17T20:45:15.1664099Z 1 with iteration mapping 2022-03-17T20:45:15.1664380Z {0: 5, 1: 8} 2022-03-17T20:45:16.2696930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:16.2697459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:20.6474636Z Running test: large_conv_model sync interval 2022-03-17T20:45:20.6474980Z 1 with iteration mapping 2022-03-17T20:45:20.6475252Z {0: 5, 1: 15} 2022-03-17T20:45:21.7371288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:21.7371797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.0900684Z Running test: small_model sync interval 2022-03-17T20:45:26.0901008Z 1 with iteration mapping 2022-03-17T20:45:26.0901505Z {0: 0, 1: 2} 2022-03-17T20:45:26.0920032Z Running test: small_model sync interval 2022-03-17T20:45:26.0920357Z 1 with iteration mapping 2022-03-17T20:45:26.0920643Z {0: 0, 1: 3} 2022-03-17T20:45:26.0938676Z Running test: small_model sync interval 2022-03-17T20:45:26.0938990Z 1 with iteration mapping 2022-03-17T20:45:26.0939242Z {0: 0, 1: 10} 2022-03-17T20:45:26.0957624Z Running test: small_model sync interval 2022-03-17T20:45:26.0957933Z 1 with iteration mapping 2022-03-17T20:45:26.0958184Z {0: 5, 1: 7} 2022-03-17T20:45:26.0999251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.0999729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.1084992Z Running test: small_model sync interval 2022-03-17T20:45:26.1085303Z 1 with iteration mapping 2022-03-17T20:45:26.1085556Z {0: 5, 1: 8} 2022-03-17T20:45:26.1126069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.1126558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.1211352Z Running test: small_model sync interval 2022-03-17T20:45:26.1211663Z 1 with iteration mapping 2022-03-17T20:45:26.1211927Z {0: 5, 1: 15} 2022-03-17T20:45:26.1252130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.1252620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.1337249Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1337597Z 1 with iteration mapping 2022-03-17T20:45:26.1337853Z {0: 0, 1: 2} 2022-03-17T20:45:26.1355656Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1356006Z 1 with iteration mapping 2022-03-17T20:45:26.1356268Z {0: 0, 1: 3} 2022-03-17T20:45:26.1373658Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1373985Z 1 with iteration mapping 2022-03-17T20:45:26.1374258Z {0: 0, 1: 10} 2022-03-17T20:45:26.1392171Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1392496Z 1 with iteration mapping 2022-03-17T20:45:26.1392768Z {0: 5, 1: 7} 2022-03-17T20:45:26.1476297Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1476628Z 1 with iteration mapping 2022-03-17T20:45:26.1476930Z {0: 5, 1: 8} 2022-03-17T20:45:26.1559244Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1559597Z 1 with iteration mapping 2022-03-17T20:45:26.1559869Z {0: 5, 1: 15} 2022-03-17T20:45:26.1642224Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1642583Z 1 with iteration mapping 2022-03-17T20:45:26.1642836Z {0: 0, 1: 2} 2022-03-17T20:45:26.1659788Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1660136Z 1 with iteration mapping 2022-03-17T20:45:26.1660615Z {0: 0, 1: 3} 2022-03-17T20:45:26.1678030Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1678359Z 1 with iteration mapping 2022-03-17T20:45:26.1678625Z {0: 0, 1: 10} 2022-03-17T20:45:26.1696198Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1696517Z 1 with iteration mapping 2022-03-17T20:45:26.1696792Z {0: 5, 1: 7} 2022-03-17T20:45:26.1781150Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1781498Z 1 with iteration mapping 2022-03-17T20:45:26.1781765Z {0: 5, 1: 8} 2022-03-17T20:45:26.1872347Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T20:45:26.1872698Z 1 with iteration mapping 2022-03-17T20:45:26.1872962Z {0: 5, 1: 15} 2022-03-17T20:45:26.1956770Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:26.1957107Z 1 with iteration mapping 2022-03-17T20:45:26.1957359Z {0: 0, 1: 2} 2022-03-17T20:45:26.1975213Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:26.1975523Z 1 with iteration mapping 2022-03-17T20:45:26.1975793Z {0: 0, 1: 3} 2022-03-17T20:45:26.1994258Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:26.1994566Z 1 with iteration mapping 2022-03-17T20:45:26.1994842Z {0: 0, 1: 10} 2022-03-17T20:45:26.2012865Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:26.2013199Z 1 with iteration mapping 2022-03-17T20:45:26.2013474Z {0: 5, 1: 7} 2022-03-17T20:45:26.2053779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2054275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2139149Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:26.2139463Z 1 with iteration mapping 2022-03-17T20:45:26.2139737Z {0: 5, 1: 8} 2022-03-17T20:45:26.2180139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2180633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2265997Z Running test: small_model_allreduce_hook sync interval 2022-03-17T20:45:26.2266319Z 1 with iteration mapping 2022-03-17T20:45:26.2266592Z {0: 5, 1: 15} 2022-03-17T20:45:26.2307101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2307740Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2392355Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:26.2392668Z 1 with iteration mapping 2022-03-17T20:45:26.2392944Z {0: 0, 1: 2} 2022-03-17T20:45:26.2410969Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:26.2411311Z 1 with iteration mapping 2022-03-17T20:45:26.2411581Z {0: 0, 1: 3} 2022-03-17T20:45:26.2429862Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:26.2430296Z 1 with iteration mapping 2022-03-17T20:45:26.2430562Z {0: 0, 1: 10} 2022-03-17T20:45:26.2448507Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:26.2448837Z 1 with iteration mapping 2022-03-17T20:45:26.2449086Z {0: 5, 1: 7} 2022-03-17T20:45:26.2489475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2489964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2574687Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:26.2575022Z 1 with iteration mapping 2022-03-17T20:45:26.2575273Z {0: 5, 1: 8} 2022-03-17T20:45:26.2615374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2615867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2700845Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T20:45:26.2701184Z 1 with iteration mapping 2022-03-17T20:45:26.2701458Z {0: 5, 1: 15} 2022-03-17T20:45:26.2741587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.2742081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:26.8016702Z ok (122.664s) 2022-03-17T20:45:26.8016936Z 2022-03-17T20:45:26.8017357Z ---------------------------------------------------------------------- 2022-03-17T20:45:26.8017694Z Ran 1 test in 122.664s 2022-03-17T20:45:26.8017839Z 2022-03-17T20:45:26.8017936Z OK 2022-03-17T20:45:26.8018068Z 2022-03-17T20:45:26.8018201Z Generating XML reports... 2022-03-17T20:45:26.8063238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204324.xml 2022-03-17T20:45:28.1055587Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:28.1071187Z 2022-03-17T20:45:28.1071409Z Running tests... 2022-03-17T20:45:28.1071832Z ---------------------------------------------------------------------- 2022-03-17T20:45:29.6443736Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:45:29.6869516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14681 2022-03-17T20:45:29.6995299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14682 2022-03-17T20:45:30.7405984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:30.7474333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:30.7475371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:30.7507590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:30.7514291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:30.8489719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:32.4959458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:32.4959994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:32.5074594Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:32.5075647Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:32.5076700Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:32.5077525Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:32.5175332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:32.5175835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:32.5223414Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:45:32.5224160Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:45:32.8087987Z ok (4.701s) 2022-03-17T20:45:32.8088244Z 2022-03-17T20:45:32.8088666Z ---------------------------------------------------------------------- 2022-03-17T20:45:32.8088994Z Ran 1 test in 4.702s 2022-03-17T20:45:32.8089157Z 2022-03-17T20:45:32.8089218Z OK 2022-03-17T20:45:32.8089340Z 2022-03-17T20:45:32.8089456Z Generating XML reports... 2022-03-17T20:45:32.8134198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204528.xml 2022-03-17T20:45:34.1147046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:34.1162323Z 2022-03-17T20:45:34.1162713Z Running tests... 2022-03-17T20:45:34.1163231Z ---------------------------------------------------------------------- 2022-03-17T20:45:35.6579815Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:45:35.7004861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14807 2022-03-17T20:45:35.7132483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14808 2022-03-17T20:45:36.8083705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:36.8149196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:36.8150453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:36.8184771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:36.8191250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:36.9164080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:39.8230889Z ok (5.706s) 2022-03-17T20:45:39.8231119Z 2022-03-17T20:45:39.8231752Z ---------------------------------------------------------------------- 2022-03-17T20:45:39.8232101Z Ran 1 test in 5.707s 2022-03-17T20:45:39.8232266Z 2022-03-17T20:45:39.8232359Z OK 2022-03-17T20:45:39.8233647Z 2022-03-17T20:45:39.8234051Z Generating XML reports... 2022-03-17T20:45:39.8276687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204534.xml 2022-03-17T20:45:41.1292150Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:41.1307641Z 2022-03-17T20:45:41.1307989Z Running tests... 2022-03-17T20:45:41.1308424Z ---------------------------------------------------------------------- 2022-03-17T20:45:42.6695190Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:45:42.7117082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14934 2022-03-17T20:45:42.7243665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14935 2022-03-17T20:45:43.8172169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:43.8339610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:43.8340722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:43.8374481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:43.8381130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:43.8384370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:45:43.9352168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:43.9355560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:45:43.9356308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:45:43.9402990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:45:44.2292688Z ok (3.098s) 2022-03-17T20:45:44.2292851Z 2022-03-17T20:45:44.2293308Z ---------------------------------------------------------------------- 2022-03-17T20:45:44.2293638Z Ran 1 test in 3.098s 2022-03-17T20:45:44.2293803Z 2022-03-17T20:45:44.2293876Z OK 2022-03-17T20:45:44.2294014Z 2022-03-17T20:45:44.2294153Z Generating XML reports... 2022-03-17T20:45:44.2340472Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204541.xml 2022-03-17T20:45:45.5207013Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:45.5222171Z 2022-03-17T20:45:45.5222563Z Running tests... 2022-03-17T20:45:45.5223006Z ---------------------------------------------------------------------- 2022-03-17T20:45:47.0760970Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:45:47.1182690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15051 2022-03-17T20:45:47.1312431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15052 2022-03-17T20:45:48.1640251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:48.2073095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:48.2073885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:48.2146421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:48.2153205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:48.2156306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:45:48.3083981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:48.3087511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:45:48.3088225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:45:48.3178674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:45:48.5360147Z ok (3.013s) 2022-03-17T20:45:48.5360504Z 2022-03-17T20:45:48.5360897Z ---------------------------------------------------------------------- 2022-03-17T20:45:48.5361210Z Ran 1 test in 3.014s 2022-03-17T20:45:48.5361353Z 2022-03-17T20:45:48.5361444Z OK 2022-03-17T20:45:48.5361579Z 2022-03-17T20:45:48.5361692Z Generating XML reports... 2022-03-17T20:45:48.5405817Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204545.xml 2022-03-17T20:45:49.8152955Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:49.8168008Z 2022-03-17T20:45:49.8168331Z Running tests... 2022-03-17T20:45:49.8168819Z ---------------------------------------------------------------------- 2022-03-17T20:45:51.3615241Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:45:51.4041740Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15168 2022-03-17T20:45:51.4169758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15169 2022-03-17T20:45:52.4618410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:52.4709345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:52.4710532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:52.4722849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:52.4729676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:52.5725105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:45:54.1868128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:54.1868658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:45:54.1926266Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:45:54.1927847Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:45:54.5248555Z ok (4.708s) 2022-03-17T20:45:54.5248751Z 2022-03-17T20:45:54.5249362Z ---------------------------------------------------------------------- 2022-03-17T20:45:54.5249710Z Ran 1 test in 4.708s 2022-03-17T20:45:54.5249880Z 2022-03-17T20:45:54.5249960Z OK 2022-03-17T20:45:54.5250100Z 2022-03-17T20:45:54.5250210Z Generating XML reports... 2022-03-17T20:45:54.5295462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204549.xml 2022-03-17T20:45:55.8189944Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:45:55.8204987Z 2022-03-17T20:45:55.8205380Z Running tests... 2022-03-17T20:45:55.8205875Z ---------------------------------------------------------------------- 2022-03-17T20:45:57.3499627Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:45:57.3922139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15294 2022-03-17T20:45:57.4050677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15295 2022-03-17T20:45:58.4819992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:45:58.4821201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:45:58.4822431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:58.4922185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:45:58.4928728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:45:58.5836744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:00.2141732Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:46:00.2285376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:46:00.2285911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:46:00.5129600Z ok (4.692s) 2022-03-17T20:46:00.5129833Z 2022-03-17T20:46:00.5130260Z ---------------------------------------------------------------------- 2022-03-17T20:46:00.5130612Z Ran 1 test in 4.692s 2022-03-17T20:46:00.5130717Z 2022-03-17T20:46:00.5130811Z OK 2022-03-17T20:46:00.5130948Z 2022-03-17T20:46:00.5131081Z Generating XML reports... 2022-03-17T20:46:00.5176388Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204555.xml 2022-03-17T20:46:01.8252378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:01.8265472Z 2022-03-17T20:46:01.8265614Z Running tests... 2022-03-17T20:46:01.8266353Z ---------------------------------------------------------------------- 2022-03-17T20:46:03.3962789Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:03.4405167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15420 2022-03-17T20:46:03.4533067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15421 2022-03-17T20:46:04.5228691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:04.5445889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:04.5446700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:04.5535000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:04.5541739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:04.6458080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:04.8578124Z ok (3.031s) 2022-03-17T20:46:04.8578323Z 2022-03-17T20:46:04.8578706Z ---------------------------------------------------------------------- 2022-03-17T20:46:04.8579037Z Ran 1 test in 3.031s 2022-03-17T20:46:04.8579226Z 2022-03-17T20:46:04.8579240Z OK 2022-03-17T20:46:04.8579356Z 2022-03-17T20:46:04.8579490Z Generating XML reports... 2022-03-17T20:46:04.8623886Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204601.xml 2022-03-17T20:46:06.1297929Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:06.1312904Z 2022-03-17T20:46:06.1313109Z Running tests... 2022-03-17T20:46:06.1314157Z ---------------------------------------------------------------------- 2022-03-17T20:46:06.1332421Z test_gather (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:06.1332879Z 2022-03-17T20:46:06.1333282Z ---------------------------------------------------------------------- 2022-03-17T20:46:06.1333601Z Ran 1 test in 0.002s 2022-03-17T20:46:06.1333766Z 2022-03-17T20:46:06.1333879Z OK (skipped=1) 2022-03-17T20:46:06.1334038Z 2022-03-17T20:46:06.1334143Z Generating XML reports... 2022-03-17T20:46:06.1377397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204606.xml 2022-03-17T20:46:07.2830131Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:07.2845835Z 2022-03-17T20:46:07.2845975Z Running tests... 2022-03-17T20:46:07.2846623Z ---------------------------------------------------------------------- 2022-03-17T20:46:07.2874240Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:46:07.2874496Z 2022-03-17T20:46:07.2874849Z ---------------------------------------------------------------------- 2022-03-17T20:46:07.2875191Z Ran 1 test in 0.003s 2022-03-17T20:46:07.2875359Z 2022-03-17T20:46:07.2875425Z OK (skipped=1) 2022-03-17T20:46:07.2875568Z 2022-03-17T20:46:07.2875696Z Generating XML reports... 2022-03-17T20:46:07.2920318Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204607.xml 2022-03-17T20:46:08.4445024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:08.4460763Z 2022-03-17T20:46:08.4460883Z Running tests... 2022-03-17T20:46:08.4461629Z ---------------------------------------------------------------------- 2022-03-17T20:46:09.9907928Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:10.0347757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15603 2022-03-17T20:46:10.0477987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15604 2022-03-17T20:46:11.1434640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:11.1540661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:11.1541736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:11.1637609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:11.1643849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:11.2556267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:15.2594034Z ok (6.813s) 2022-03-17T20:46:15.2594259Z 2022-03-17T20:46:15.2594718Z ---------------------------------------------------------------------- 2022-03-17T20:46:15.2595040Z Ran 1 test in 6.813s 2022-03-17T20:46:15.2595208Z 2022-03-17T20:46:15.2595301Z OK 2022-03-17T20:46:15.2595617Z 2022-03-17T20:46:15.2595749Z Generating XML reports... 2022-03-17T20:46:15.2639846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204608.xml 2022-03-17T20:46:16.5664495Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:16.5679991Z 2022-03-17T20:46:16.5680133Z Running tests... 2022-03-17T20:46:16.5681406Z ---------------------------------------------------------------------- 2022-03-17T20:46:16.5700033Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:16.5700680Z 2022-03-17T20:46:16.5700998Z ---------------------------------------------------------------------- 2022-03-17T20:46:16.5701344Z Ran 1 test in 0.002s 2022-03-17T20:46:16.5701511Z 2022-03-17T20:46:16.5701587Z OK (skipped=1) 2022-03-17T20:46:16.5701724Z 2022-03-17T20:46:16.5701851Z Generating XML reports... 2022-03-17T20:46:16.5746166Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204616.xml 2022-03-17T20:46:17.7271915Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:17.7288193Z 2022-03-17T20:46:17.7288500Z Running tests... 2022-03-17T20:46:17.7288949Z ---------------------------------------------------------------------- 2022-03-17T20:46:17.7308924Z test_gather_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:46:17.7309401Z 2022-03-17T20:46:17.7309778Z ---------------------------------------------------------------------- 2022-03-17T20:46:17.7310088Z Ran 1 test in 0.002s 2022-03-17T20:46:17.7310160Z 2022-03-17T20:46:17.7310273Z OK (skipped=1) 2022-03-17T20:46:17.7310427Z 2022-03-17T20:46:17.7310533Z Generating XML reports... 2022-03-17T20:46:17.7354717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204617.xml 2022-03-17T20:46:18.8885750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:18.8901117Z 2022-03-17T20:46:18.8901440Z Running tests... 2022-03-17T20:46:18.8901862Z ---------------------------------------------------------------------- 2022-03-17T20:46:20.4399823Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:20.4825396Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15801 2022-03-17T20:46:20.4950595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15802 2022-03-17T20:46:21.5899843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:21.6009872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:21.6010690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:21.6102452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:21.6109223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:21.7025208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:24.4045189Z ok (5.514s) 2022-03-17T20:46:24.4045602Z 2022-03-17T20:46:24.4046105Z ---------------------------------------------------------------------- 2022-03-17T20:46:24.4046395Z Ran 1 test in 5.514s 2022-03-17T20:46:24.4046539Z 2022-03-17T20:46:24.4046618Z OK 2022-03-17T20:46:24.4047413Z 2022-03-17T20:46:24.4047574Z Generating XML reports... 2022-03-17T20:46:24.4091452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204618.xml 2022-03-17T20:46:25.7088598Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:25.7105259Z 2022-03-17T20:46:25.7105424Z Running tests... 2022-03-17T20:46:25.7106484Z ---------------------------------------------------------------------- 2022-03-17T20:46:27.2548917Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:27.2972603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15925 2022-03-17T20:46:27.3098968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15926 2022-03-17T20:46:28.3535000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:28.3859534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:28.3861365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:28.3939524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:28.3946597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:28.4872288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:28.4986225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:46:28.4987067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:46:28.4988201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:28.4989368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:31.0002907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:46:31.0004061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:46:31.0005407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:46:31.0006831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:46:31.0459041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:46:31.0460144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:46:31.0461847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:46:31.0463237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:46:31.5196425Z ok (5.809s) 2022-03-17T20:46:31.5196700Z 2022-03-17T20:46:31.5197046Z ---------------------------------------------------------------------- 2022-03-17T20:46:31.5197433Z Ran 1 test in 5.809s 2022-03-17T20:46:31.5197583Z 2022-03-17T20:46:31.5197673Z OK 2022-03-17T20:46:31.5197794Z 2022-03-17T20:46:31.5198126Z Generating XML reports... 2022-03-17T20:46:31.5243016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204625.xml 2022-03-17T20:46:32.8173594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:32.8188938Z 2022-03-17T20:46:32.8189331Z Running tests... 2022-03-17T20:46:32.8189793Z ---------------------------------------------------------------------- 2022-03-17T20:46:34.3626725Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:34.4054890Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16088 2022-03-17T20:46:34.4182269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16089 2022-03-17T20:46:35.4821972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:35.5041946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:35.5042696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:35.5128082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:35.5134281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:35.5137325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:46:35.6053479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:35.6059505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:46:35.6061151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:35.6157670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:35.8231676Z ok (3.004s) 2022-03-17T20:46:35.8232163Z 2022-03-17T20:46:35.8232789Z ---------------------------------------------------------------------- 2022-03-17T20:46:35.8233360Z Ran 1 test in 3.004s 2022-03-17T20:46:35.8233670Z 2022-03-17T20:46:35.8233824Z OK 2022-03-17T20:46:35.8234068Z 2022-03-17T20:46:35.8234308Z Generating XML reports... 2022-03-17T20:46:35.8278963Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204632.xml 2022-03-17T20:46:37.1017768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:37.1033042Z 2022-03-17T20:46:37.1033300Z Running tests... 2022-03-17T20:46:37.1033772Z ---------------------------------------------------------------------- 2022-03-17T20:46:38.6433171Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:38.6858913Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16205 2022-03-17T20:46:38.6987021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16206 2022-03-17T20:46:39.7743081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:39.7943559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:39.7944355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:39.7945306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:39.7952714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:39.7953657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:42.4076785Z ok (5.304s) 2022-03-17T20:46:42.4077226Z 2022-03-17T20:46:42.4077863Z ---------------------------------------------------------------------- 2022-03-17T20:46:42.4078509Z Ran 1 test in 5.304s 2022-03-17T20:46:42.4078792Z 2022-03-17T20:46:42.4078958Z OK 2022-03-17T20:46:42.4079196Z 2022-03-17T20:46:42.4079439Z Generating XML reports... 2022-03-17T20:46:42.4125881Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204637.xml 2022-03-17T20:46:43.7047978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:43.7063575Z 2022-03-17T20:46:43.7063735Z Running tests... 2022-03-17T20:46:43.7064123Z ---------------------------------------------------------------------- 2022-03-17T20:46:45.2482176Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:45.2910239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16328 2022-03-17T20:46:45.3037525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16329 2022-03-17T20:46:46.3502626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:46.3923169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:46.3924016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:46.4009799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:46.4016502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:46.4934722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:46.8085953Z ok (3.102s) 2022-03-17T20:46:46.8086345Z 2022-03-17T20:46:46.8087055Z ---------------------------------------------------------------------- 2022-03-17T20:46:46.8087512Z Ran 1 test in 3.102s 2022-03-17T20:46:46.8087682Z 2022-03-17T20:46:46.8087772Z OK 2022-03-17T20:46:46.8087906Z 2022-03-17T20:46:46.8088036Z Generating XML reports... 2022-03-17T20:46:46.8131475Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204643.xml 2022-03-17T20:46:48.0393452Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:48.0409040Z 2022-03-17T20:46:48.0409177Z Running tests... 2022-03-17T20:46:48.0410509Z ---------------------------------------------------------------------- 2022-03-17T20:46:49.5775258Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:49.6200126Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16441 2022-03-17T20:46:49.6329158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16442 2022-03-17T20:46:50.6988831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:50.7242198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:50.7243204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:50.7292940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:50.7299752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:50.7303927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:46:50.8255782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:50.8260298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:46:50.8261236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:50.8322497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:51.0375380Z ok (2.996s) 2022-03-17T20:46:51.0375637Z 2022-03-17T20:46:51.0375976Z ---------------------------------------------------------------------- 2022-03-17T20:46:51.0376294Z Ran 1 test in 2.997s 2022-03-17T20:46:51.0376446Z 2022-03-17T20:46:51.0376763Z OK 2022-03-17T20:46:51.0376914Z 2022-03-17T20:46:51.0377029Z Generating XML reports... 2022-03-17T20:46:51.0422101Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204648.xml 2022-03-17T20:46:52.3188265Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:52.3203603Z 2022-03-17T20:46:52.3204052Z Running tests... 2022-03-17T20:46:52.3204553Z ---------------------------------------------------------------------- 2022-03-17T20:46:53.8797466Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:53.9225900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16558 2022-03-17T20:46:53.9356600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16559 2022-03-17T20:46:55.0056443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:55.0433420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:55.0434736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:55.0461165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:55.0468286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:55.0471546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:46:55.1445777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:46:55.1449825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:46:55.1450859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:55.1489857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:46:55.4407778Z ok (3.120s) 2022-03-17T20:46:55.4407907Z 2022-03-17T20:46:55.4408378Z ---------------------------------------------------------------------- 2022-03-17T20:46:55.4408720Z Ran 1 test in 3.120s 2022-03-17T20:46:55.4408879Z 2022-03-17T20:46:55.4408938Z OK 2022-03-17T20:46:55.4409082Z 2022-03-17T20:46:55.4409261Z Generating XML reports... 2022-03-17T20:46:55.4454124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204652.xml 2022-03-17T20:46:56.7195834Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:46:56.7210777Z 2022-03-17T20:46:56.7211053Z Running tests... 2022-03-17T20:46:56.7211485Z ---------------------------------------------------------------------- 2022-03-17T20:46:58.2688825Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:46:58.3114800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16675 2022-03-17T20:46:58.3241947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16676 2022-03-17T20:46:59.3639958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:46:59.3719206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:46:59.3719946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:59.3741264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:46:59.3748488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:46:59.4733787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:01.4323534Z ok (4.711s) 2022-03-17T20:47:01.4324009Z 2022-03-17T20:47:01.4324629Z ---------------------------------------------------------------------- 2022-03-17T20:47:01.4325239Z Ran 1 test in 4.711s 2022-03-17T20:47:01.4325553Z 2022-03-17T20:47:01.4325712Z OK 2022-03-17T20:47:01.4325934Z 2022-03-17T20:47:01.4326174Z Generating XML reports... 2022-03-17T20:47:01.4372519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204656.xml 2022-03-17T20:47:02.7368532Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:02.7383787Z 2022-03-17T20:47:02.7384045Z Running tests... 2022-03-17T20:47:02.7384619Z ---------------------------------------------------------------------- 2022-03-17T20:47:02.7411238Z test_irecv (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:47:02.7411758Z 2022-03-17T20:47:02.7412502Z ---------------------------------------------------------------------- 2022-03-17T20:47:02.7412980Z Ran 1 test in 0.003s 2022-03-17T20:47:02.7413149Z 2022-03-17T20:47:02.7413274Z OK (skipped=1) 2022-03-17T20:47:02.7413431Z 2022-03-17T20:47:02.7413537Z Generating XML reports... 2022-03-17T20:47:02.7456119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204702.xml 2022-03-17T20:47:03.8873248Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:03.8888793Z 2022-03-17T20:47:03.8889259Z Running tests... 2022-03-17T20:47:03.8889929Z ---------------------------------------------------------------------- 2022-03-17T20:47:03.8908911Z test_isend (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:47:03.8909122Z 2022-03-17T20:47:03.8909495Z ---------------------------------------------------------------------- 2022-03-17T20:47:03.8910048Z Ran 1 test in 0.002s 2022-03-17T20:47:03.8910223Z 2022-03-17T20:47:03.8910344Z OK (skipped=1) 2022-03-17T20:47:03.8910448Z 2022-03-17T20:47:03.8910581Z Generating XML reports... 2022-03-17T20:47:03.8954639Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204703.xml 2022-03-17T20:47:05.0467283Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:05.0481965Z 2022-03-17T20:47:05.0482409Z Running tests... 2022-03-17T20:47:05.0482911Z ---------------------------------------------------------------------- 2022-03-17T20:47:05.0501581Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:47:05.0501812Z 2022-03-17T20:47:05.0502160Z ---------------------------------------------------------------------- 2022-03-17T20:47:05.0502497Z Ran 1 test in 0.002s 2022-03-17T20:47:05.0502683Z 2022-03-17T20:47:05.0504159Z OK (skipped=1) 2022-03-17T20:47:05.0504357Z 2022-03-17T20:47:05.0504451Z Generating XML reports... 2022-03-17T20:47:05.0548653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204705.xml 2022-03-17T20:47:06.1667618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:06.1682421Z 2022-03-17T20:47:06.1682604Z Running tests... 2022-03-17T20:47:06.1683055Z ---------------------------------------------------------------------- 2022-03-17T20:47:06.1703604Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:47:06.1703866Z 2022-03-17T20:47:06.1704162Z ---------------------------------------------------------------------- 2022-03-17T20:47:06.1704477Z Ran 1 test in 0.002s 2022-03-17T20:47:06.1704654Z 2022-03-17T20:47:06.1705213Z OK (skipped=1) 2022-03-17T20:47:06.1705482Z 2022-03-17T20:47:06.1705894Z Generating XML reports... 2022-03-17T20:47:06.1749386Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204706.xml 2022-03-17T20:47:07.3156157Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:07.3171509Z 2022-03-17T20:47:07.3171872Z Running tests... 2022-03-17T20:47:07.3172356Z ---------------------------------------------------------------------- 2022-03-17T20:47:08.8670367Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:47:08.9098184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16941 2022-03-17T20:47:08.9227487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16942 2022-03-17T20:47:09.9944360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:10.0153154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:10.0153947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:10.0248791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:10.0255369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:10.0258538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:47:10.1164389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:10.1168641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:47:10.1169524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:47:10.1172507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:47:10.1380428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:47:10.1381275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:47:10.1381950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:47:10.1382625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:47:12.5662082Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T20:47:28.3592754Z ok (21.042s) 2022-03-17T20:47:28.3592979Z 2022-03-17T20:47:28.3595517Z ---------------------------------------------------------------------- 2022-03-17T20:47:28.3596191Z Ran 1 test in 21.042s 2022-03-17T20:47:28.3596374Z 2022-03-17T20:47:28.3596466Z OK 2022-03-17T20:47:28.3596600Z 2022-03-17T20:47:28.3596717Z Generating XML reports... 2022-03-17T20:47:28.3639618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204707.xml 2022-03-17T20:47:29.6646528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:29.6661804Z 2022-03-17T20:47:29.6662219Z Running tests... 2022-03-17T20:47:29.6662647Z ---------------------------------------------------------------------- 2022-03-17T20:47:31.2149575Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:47:31.2576434Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17072 2022-03-17T20:47:31.2705525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17073 2022-03-17T20:47:32.3241036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:47:32.3468903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:47:32.3470393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:32.3545143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:47:32.3551531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:47:32.3554828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:47:32.4480873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:47:32.4485203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:47:32.4486465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:47:32.4573406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:47:32.4696212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:47:32.4697334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:47:32.4698173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:47:32.4698839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:47:34.8878278Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T20:47:34.8878832Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T20:47:50.3045683Z ok (20.638s) 2022-03-17T20:47:50.3045909Z 2022-03-17T20:47:50.3046321Z ---------------------------------------------------------------------- 2022-03-17T20:47:50.3046640Z Ran 1 test in 20.638s 2022-03-17T20:47:50.3046780Z 2022-03-17T20:47:50.3049678Z OK 2022-03-17T20:47:50.3050032Z 2022-03-17T20:47:50.3050313Z Generating XML reports... 2022-03-17T20:47:50.3102305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204729.xml 2022-03-17T20:47:51.6089690Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:51.6106473Z 2022-03-17T20:47:51.6106761Z Running tests... 2022-03-17T20:47:51.6107173Z ---------------------------------------------------------------------- 2022-03-17T20:47:51.6136672Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:47:51.6136969Z 2022-03-17T20:47:51.6137514Z ---------------------------------------------------------------------- 2022-03-17T20:47:51.6137870Z Ran 1 test in 0.003s 2022-03-17T20:47:51.6138016Z 2022-03-17T20:47:51.6138130Z OK (skipped=1) 2022-03-17T20:47:51.6138574Z 2022-03-17T20:47:51.6138702Z Generating XML reports... 2022-03-17T20:47:51.6189615Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204751.xml 2022-03-17T20:47:52.7608652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:52.7624441Z 2022-03-17T20:47:52.7624751Z Running tests... 2022-03-17T20:47:52.7625211Z ---------------------------------------------------------------------- 2022-03-17T20:47:52.7660562Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:47:52.7662610Z 2022-03-17T20:47:52.7663765Z ---------------------------------------------------------------------- 2022-03-17T20:47:52.7664166Z Ran 1 test in 0.004s 2022-03-17T20:47:52.7664346Z 2022-03-17T20:47:52.7664458Z OK (skipped=1) 2022-03-17T20:47:52.7664598Z 2022-03-17T20:47:52.7664729Z Generating XML reports... 2022-03-17T20:47:52.7712983Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204752.xml 2022-03-17T20:47:53.9116596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:53.9132309Z 2022-03-17T20:47:53.9132543Z Running tests... 2022-03-17T20:47:53.9132975Z ---------------------------------------------------------------------- 2022-03-17T20:47:53.9159318Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:47:53.9159576Z 2022-03-17T20:47:53.9159930Z ---------------------------------------------------------------------- 2022-03-17T20:47:53.9160250Z Ran 1 test in 0.003s 2022-03-17T20:47:53.9160433Z 2022-03-17T20:47:53.9160545Z OK (skipped=1) 2022-03-17T20:47:53.9160705Z 2022-03-17T20:47:53.9160807Z Generating XML reports... 2022-03-17T20:47:53.9212899Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204753.xml 2022-03-17T20:47:55.0668772Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:55.0684888Z 2022-03-17T20:47:55.0685323Z Running tests... 2022-03-17T20:47:55.0685807Z ---------------------------------------------------------------------- 2022-03-17T20:47:55.0714231Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:47:55.0714473Z 2022-03-17T20:47:55.0714816Z ---------------------------------------------------------------------- 2022-03-17T20:47:55.0715079Z Ran 1 test in 0.003s 2022-03-17T20:47:55.0715246Z 2022-03-17T20:47:55.0716912Z OK (skipped=1) 2022-03-17T20:47:55.0717356Z 2022-03-17T20:47:55.0717739Z Generating XML reports... 2022-03-17T20:47:55.0767325Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204755.xml 2022-03-17T20:47:56.2282358Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:56.2298252Z 2022-03-17T20:47:56.2298348Z Running tests... 2022-03-17T20:47:56.2299052Z ---------------------------------------------------------------------- 2022-03-17T20:47:56.2327143Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:47:56.2327483Z 2022-03-17T20:47:56.2327725Z ---------------------------------------------------------------------- 2022-03-17T20:47:56.2328063Z Ran 1 test in 0.003s 2022-03-17T20:47:56.2328226Z 2022-03-17T20:47:56.2328318Z OK (skipped=1) 2022-03-17T20:47:56.2328532Z 2022-03-17T20:47:56.2328660Z Generating XML reports... 2022-03-17T20:47:56.2380186Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204756.xml 2022-03-17T20:47:57.3778327Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:47:57.3793953Z 2022-03-17T20:47:57.3794306Z Running tests... 2022-03-17T20:47:57.3795127Z ---------------------------------------------------------------------- 2022-03-17T20:47:58.9263408Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:47:58.9690498Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17378 2022-03-17T20:47:58.9823732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17379 2022-03-17T20:48:00.0312461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:00.0313020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:00.0314016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:00.0314744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:00.0320337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:00.0322181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:01.6895756Z ok (4.310s) 2022-03-17T20:48:01.6895975Z 2022-03-17T20:48:01.6896376Z ---------------------------------------------------------------------- 2022-03-17T20:48:01.6896684Z Ran 1 test in 4.310s 2022-03-17T20:48:01.6896852Z 2022-03-17T20:48:01.6896945Z OK 2022-03-17T20:48:01.6897087Z 2022-03-17T20:48:01.6897223Z Generating XML reports... 2022-03-17T20:48:01.6941537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204757.xml 2022-03-17T20:48:02.9965712Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:02.9981212Z 2022-03-17T20:48:02.9981487Z Running tests... 2022-03-17T20:48:02.9981797Z ---------------------------------------------------------------------- 2022-03-17T20:48:04.5363596Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:04.5787161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17500 2022-03-17T20:48:04.5912688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17501 2022-03-17T20:48:05.6570798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:05.6760770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:05.6761845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:05.6773220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:05.6779714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:05.7776847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:07.3986820Z ok (4.400s) 2022-03-17T20:48:07.3987164Z 2022-03-17T20:48:07.3987518Z ---------------------------------------------------------------------- 2022-03-17T20:48:07.3987863Z Ran 1 test in 4.400s 2022-03-17T20:48:07.3988012Z 2022-03-17T20:48:07.3988106Z OK 2022-03-17T20:48:07.3988250Z 2022-03-17T20:48:07.3988390Z Generating XML reports... 2022-03-17T20:48:07.4033034Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204802.xml 2022-03-17T20:48:08.6985036Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:08.7000490Z 2022-03-17T20:48:08.7000924Z Running tests... 2022-03-17T20:48:08.7001595Z ---------------------------------------------------------------------- 2022-03-17T20:48:10.2509137Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:10.2935206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17622 2022-03-17T20:48:10.3067841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17623 2022-03-17T20:48:11.3612567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:11.3838705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:11.3839427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:11.3916260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:11.3922814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:11.4853920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:14.0158282Z ok (5.315s) 2022-03-17T20:48:14.0159953Z 2022-03-17T20:48:14.0160707Z ---------------------------------------------------------------------- 2022-03-17T20:48:14.0161334Z Ran 1 test in 5.316s 2022-03-17T20:48:14.0161512Z 2022-03-17T20:48:14.0161615Z OK 2022-03-17T20:48:14.0161719Z 2022-03-17T20:48:14.0161807Z Generating XML reports... 2022-03-17T20:48:14.0207803Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204808.xml 2022-03-17T20:48:15.3186348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:15.3201227Z 2022-03-17T20:48:15.3201635Z Running tests... 2022-03-17T20:48:15.3202107Z ---------------------------------------------------------------------- 2022-03-17T20:48:16.8784981Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:16.9210817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17745 2022-03-17T20:48:16.9340684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17746 2022-03-17T20:48:18.0056652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:18.0395156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:18.0395881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:18.0461676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:18.0469268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:18.1411067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:19.8424885Z ok (4.522s) 2022-03-17T20:48:19.8425188Z 2022-03-17T20:48:19.8425588Z ---------------------------------------------------------------------- 2022-03-17T20:48:19.8425931Z Ran 1 test in 4.522s 2022-03-17T20:48:19.8426095Z 2022-03-17T20:48:19.8426168Z OK 2022-03-17T20:48:19.8426302Z 2022-03-17T20:48:19.8426440Z Generating XML reports... 2022-03-17T20:48:19.8470045Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204815.xml 2022-03-17T20:48:21.1031375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:21.1046773Z 2022-03-17T20:48:21.1047225Z Running tests... 2022-03-17T20:48:21.1047766Z ---------------------------------------------------------------------- 2022-03-17T20:48:22.6421593Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:22.6872124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17867 2022-03-17T20:48:22.7010481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17868 2022-03-17T20:48:23.7509491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:23.7923383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:23.7924410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:23.8015650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:23.8023259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:23.8934627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:26.2110996Z ok (5.106s) 2022-03-17T20:48:26.2111283Z 2022-03-17T20:48:26.2111663Z ---------------------------------------------------------------------- 2022-03-17T20:48:26.2112009Z Ran 1 test in 5.106s 2022-03-17T20:48:26.2112179Z 2022-03-17T20:48:26.2112273Z OK 2022-03-17T20:48:26.2112392Z 2022-03-17T20:48:26.2113460Z Generating XML reports... 2022-03-17T20:48:26.2158008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204821.xml 2022-03-17T20:48:27.5147985Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:27.5162751Z 2022-03-17T20:48:27.5162988Z Running tests... 2022-03-17T20:48:27.5163650Z ---------------------------------------------------------------------- 2022-03-17T20:48:27.5187222Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:48:27.5187469Z 2022-03-17T20:48:27.5187782Z ---------------------------------------------------------------------- 2022-03-17T20:48:27.5188299Z Ran 1 test in 0.002s 2022-03-17T20:48:27.5188512Z 2022-03-17T20:48:27.5188646Z OK (skipped=1) 2022-03-17T20:48:27.5188772Z 2022-03-17T20:48:27.5188900Z Generating XML reports... 2022-03-17T20:48:27.5232959Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204827.xml 2022-03-17T20:48:28.6565464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:28.6580135Z 2022-03-17T20:48:28.6580399Z Running tests... 2022-03-17T20:48:28.6581249Z ---------------------------------------------------------------------- 2022-03-17T20:48:28.6608592Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:48:28.6608911Z 2022-03-17T20:48:28.6609179Z ---------------------------------------------------------------------- 2022-03-17T20:48:28.6609685Z Ran 1 test in 0.003s 2022-03-17T20:48:28.6609898Z 2022-03-17T20:48:28.6610025Z OK (skipped=1) 2022-03-17T20:48:28.6610178Z 2022-03-17T20:48:28.6610305Z Generating XML reports... 2022-03-17T20:48:28.6653442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204828.xml 2022-03-17T20:48:29.8079191Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:29.8094019Z 2022-03-17T20:48:29.8094293Z Running tests... 2022-03-17T20:48:29.8094865Z ---------------------------------------------------------------------- 2022-03-17T20:48:29.8119145Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:48:29.8119708Z 2022-03-17T20:48:29.8120039Z ---------------------------------------------------------------------- 2022-03-17T20:48:29.8120372Z Ran 1 test in 0.003s 2022-03-17T20:48:29.8120536Z 2022-03-17T20:48:29.8120643Z OK (skipped=1) 2022-03-17T20:48:29.8120933Z 2022-03-17T20:48:29.8121141Z Generating XML reports... 2022-03-17T20:48:29.8163984Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204829.xml 2022-03-17T20:48:30.9034487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:30.9050854Z 2022-03-17T20:48:30.9051333Z Running tests... 2022-03-17T20:48:30.9051841Z ---------------------------------------------------------------------- 2022-03-17T20:48:32.4534114Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:32.4961631Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18105 2022-03-17T20:48:32.5088815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18106 2022-03-17T20:48:33.5803398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:33.5910108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:33.5911086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:33.6006386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:33.6013004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:33.6924389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:33.8134137Z ok (2.908s) 2022-03-17T20:48:33.8134517Z 2022-03-17T20:48:33.8134973Z ---------------------------------------------------------------------- 2022-03-17T20:48:33.8135294Z Ran 1 test in 2.908s 2022-03-17T20:48:33.8135418Z 2022-03-17T20:48:33.8135509Z OK 2022-03-17T20:48:33.8135641Z 2022-03-17T20:48:33.8135752Z Generating XML reports... 2022-03-17T20:48:33.8179464Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204830.xml 2022-03-17T20:48:35.0984202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:35.0999490Z 2022-03-17T20:48:35.0999758Z Running tests... 2022-03-17T20:48:35.1000209Z ---------------------------------------------------------------------- 2022-03-17T20:48:36.6339530Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:36.6768273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18218 2022-03-17T20:48:36.6894007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18219 2022-03-17T20:48:37.7271329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:37.7325535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:37.7326339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:37.7372865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:37.7379541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:37.8338997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:37.9940856Z ok (2.894s) 2022-03-17T20:48:37.9941057Z 2022-03-17T20:48:37.9941440Z ---------------------------------------------------------------------- 2022-03-17T20:48:37.9941806Z Ran 1 test in 2.894s 2022-03-17T20:48:37.9941973Z 2022-03-17T20:48:37.9942046Z OK 2022-03-17T20:48:37.9942192Z 2022-03-17T20:48:37.9942319Z Generating XML reports... 2022-03-17T20:48:37.9988100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204835.xml 2022-03-17T20:48:39.2751318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:39.2766318Z 2022-03-17T20:48:39.2766534Z Running tests... 2022-03-17T20:48:39.2767032Z ---------------------------------------------------------------------- 2022-03-17T20:48:39.2789079Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:48:39.2789381Z 2022-03-17T20:48:39.2789655Z ---------------------------------------------------------------------- 2022-03-17T20:48:39.2789983Z Ran 1 test in 0.002s 2022-03-17T20:48:39.2790147Z 2022-03-17T20:48:39.2790241Z OK (skipped=1) 2022-03-17T20:48:39.2790640Z 2022-03-17T20:48:39.2790778Z Generating XML reports... 2022-03-17T20:48:39.2835283Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204839.xml 2022-03-17T20:48:40.4231408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:40.4246268Z 2022-03-17T20:48:40.4246569Z Running tests... 2022-03-17T20:48:40.4246988Z ---------------------------------------------------------------------- 2022-03-17T20:48:40.4267980Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:48:40.4268301Z 2022-03-17T20:48:40.4268586Z ---------------------------------------------------------------------- 2022-03-17T20:48:40.4268980Z Ran 1 test in 0.002s 2022-03-17T20:48:40.4269094Z 2022-03-17T20:48:40.4269192Z OK (skipped=1) 2022-03-17T20:48:40.4269349Z 2022-03-17T20:48:40.4269454Z Generating XML reports... 2022-03-17T20:48:40.4312928Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204840.xml 2022-03-17T20:48:41.5773543Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:41.5788973Z 2022-03-17T20:48:41.5789222Z Running tests... 2022-03-17T20:48:41.5789691Z ---------------------------------------------------------------------- 2022-03-17T20:48:43.1305022Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:43.1728611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18401 2022-03-17T20:48:43.1856403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18402 2022-03-17T20:48:44.2341818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:44.2380313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:44.2381544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:44.2443286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:44.2450319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:44.3395767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:47.2952220Z ok (5.716s) 2022-03-17T20:48:47.2952433Z 2022-03-17T20:48:47.2952837Z ---------------------------------------------------------------------- 2022-03-17T20:48:47.2953159Z Ran 1 test in 5.716s 2022-03-17T20:48:47.2953324Z 2022-03-17T20:48:47.2953414Z OK 2022-03-17T20:48:47.2953547Z 2022-03-17T20:48:47.2953661Z Generating XML reports... 2022-03-17T20:48:47.2998510Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204841.xml 2022-03-17T20:48:48.5969096Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:48.5984832Z 2022-03-17T20:48:48.5985089Z Running tests... 2022-03-17T20:48:48.5986372Z ---------------------------------------------------------------------- 2022-03-17T20:48:50.1463437Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:50.1890625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18528 2022-03-17T20:48:50.2017833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18529 2022-03-17T20:48:51.2577219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:51.2781912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:51.2782921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:51.2880842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:51.2887667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:51.3796880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:48:54.4116788Z ok (5.813s) 2022-03-17T20:48:54.4117034Z 2022-03-17T20:48:54.4117464Z ---------------------------------------------------------------------- 2022-03-17T20:48:54.4117750Z Ran 1 test in 5.813s 2022-03-17T20:48:54.4117923Z 2022-03-17T20:48:54.4117999Z OK 2022-03-17T20:48:54.4118135Z 2022-03-17T20:48:54.4118271Z Generating XML reports... 2022-03-17T20:48:54.4162757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204848.xml 2022-03-17T20:48:55.7045154Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:48:55.7060699Z 2022-03-17T20:48:55.7061300Z Running tests... 2022-03-17T20:48:55.7061756Z ---------------------------------------------------------------------- 2022-03-17T20:48:57.2572474Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:48:57.2998296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18655 2022-03-17T20:48:57.3131926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18656 2022-03-17T20:48:58.3927977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:48:58.3994590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:48:58.3996406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:58.4032052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:48:58.4038575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:48:58.5011077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:01.2223855Z ok (5.516s) 2022-03-17T20:49:01.2224123Z 2022-03-17T20:49:01.2224522Z ---------------------------------------------------------------------- 2022-03-17T20:49:01.2224862Z Ran 1 test in 5.516s 2022-03-17T20:49:01.2225002Z 2022-03-17T20:49:01.2225076Z OK 2022-03-17T20:49:01.2225212Z 2022-03-17T20:49:01.2225343Z Generating XML reports... 2022-03-17T20:49:01.2269119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204855.xml 2022-03-17T20:49:02.5449938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:02.5465812Z 2022-03-17T20:49:02.5466208Z Running tests... 2022-03-17T20:49:02.5466700Z ---------------------------------------------------------------------- 2022-03-17T20:49:04.0981585Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:49:04.1407102Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18778 2022-03-17T20:49:04.1537708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18779 2022-03-17T20:49:05.1375094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:05.1789356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:05.1790176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:05.1882872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:05.1889676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:05.2804488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:06.9137661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:06.9138184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:06.9153327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:06.9155477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:49:07.3619459Z ok (4.815s) 2022-03-17T20:49:07.3619772Z 2022-03-17T20:49:07.3620598Z ---------------------------------------------------------------------- 2022-03-17T20:49:07.3620955Z Ran 1 test in 4.815s 2022-03-17T20:49:07.3621094Z 2022-03-17T20:49:07.3621183Z OK 2022-03-17T20:49:07.3621279Z 2022-03-17T20:49:07.3621417Z Generating XML reports... 2022-03-17T20:49:07.3665646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204902.xml 2022-03-17T20:49:08.6566780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:08.6581910Z 2022-03-17T20:49:08.6582129Z Running tests... 2022-03-17T20:49:08.6582760Z ---------------------------------------------------------------------- 2022-03-17T20:49:08.6604376Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:08.6604795Z 2022-03-17T20:49:08.6605094Z ---------------------------------------------------------------------- 2022-03-17T20:49:08.6605378Z Ran 1 test in 0.002s 2022-03-17T20:49:08.6605543Z 2022-03-17T20:49:08.6605632Z OK (skipped=1) 2022-03-17T20:49:08.6605802Z 2022-03-17T20:49:08.6605926Z Generating XML reports... 2022-03-17T20:49:08.6649717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204908.xml 2022-03-17T20:49:09.8073250Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:09.8088028Z 2022-03-17T20:49:09.8088169Z Running tests... 2022-03-17T20:49:09.8088880Z ---------------------------------------------------------------------- 2022-03-17T20:49:09.8110410Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:09.8110718Z 2022-03-17T20:49:09.8111043Z ---------------------------------------------------------------------- 2022-03-17T20:49:09.8111374Z Ran 1 test in 0.002s 2022-03-17T20:49:09.8111521Z 2022-03-17T20:49:09.8111630Z OK (skipped=1) 2022-03-17T20:49:09.8111764Z 2022-03-17T20:49:09.8111897Z Generating XML reports... 2022-03-17T20:49:09.8155385Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204909.xml 2022-03-17T20:49:10.9708333Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:10.9723300Z 2022-03-17T20:49:10.9723736Z Running tests... 2022-03-17T20:49:10.9724177Z ---------------------------------------------------------------------- 2022-03-17T20:49:10.9747267Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:10.9747593Z 2022-03-17T20:49:10.9747854Z ---------------------------------------------------------------------- 2022-03-17T20:49:10.9748163Z Ran 1 test in 0.002s 2022-03-17T20:49:10.9748309Z 2022-03-17T20:49:10.9748421Z OK (skipped=1) 2022-03-17T20:49:10.9748576Z 2022-03-17T20:49:10.9748701Z Generating XML reports... 2022-03-17T20:49:10.9792115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204910.xml 2022-03-17T20:49:12.1288775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:12.1303757Z 2022-03-17T20:49:12.1304059Z Running tests... 2022-03-17T20:49:12.1304420Z ---------------------------------------------------------------------- 2022-03-17T20:49:12.1326570Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:12.1326831Z 2022-03-17T20:49:12.1327182Z ---------------------------------------------------------------------- 2022-03-17T20:49:12.1327494Z Ran 1 test in 0.002s 2022-03-17T20:49:12.1327653Z 2022-03-17T20:49:12.1327770Z OK (skipped=1) 2022-03-17T20:49:12.1327863Z 2022-03-17T20:49:12.1327971Z Generating XML reports... 2022-03-17T20:49:12.1371943Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204912.xml 2022-03-17T20:49:13.2352594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:13.2367592Z 2022-03-17T20:49:13.2367977Z Running tests... 2022-03-17T20:49:13.2368467Z ---------------------------------------------------------------------- 2022-03-17T20:49:13.2390024Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:13.2390340Z 2022-03-17T20:49:13.2390624Z ---------------------------------------------------------------------- 2022-03-17T20:49:13.2390955Z Ran 1 test in 0.002s 2022-03-17T20:49:13.2391124Z 2022-03-17T20:49:13.2391214Z OK (skipped=1) 2022-03-17T20:49:13.2391326Z 2022-03-17T20:49:13.2391451Z Generating XML reports... 2022-03-17T20:49:13.2435815Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204913.xml 2022-03-17T20:49:14.3873049Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:14.3888255Z 2022-03-17T20:49:14.3888504Z Running tests... 2022-03-17T20:49:14.3888974Z ---------------------------------------------------------------------- 2022-03-17T20:49:14.3910316Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:14.3910594Z 2022-03-17T20:49:14.3910907Z ---------------------------------------------------------------------- 2022-03-17T20:49:14.3911250Z Ran 1 test in 0.002s 2022-03-17T20:49:14.3911390Z 2022-03-17T20:49:14.3911499Z OK (skipped=1) 2022-03-17T20:49:14.3911616Z 2022-03-17T20:49:14.3914107Z Generating XML reports... 2022-03-17T20:49:14.3955934Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204914.xml 2022-03-17T20:49:15.5366747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:15.5381365Z 2022-03-17T20:49:15.5381595Z Running tests... 2022-03-17T20:49:15.5382073Z ---------------------------------------------------------------------- 2022-03-17T20:49:15.5405042Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:15.5405548Z 2022-03-17T20:49:15.5405943Z ---------------------------------------------------------------------- 2022-03-17T20:49:15.5406280Z Ran 1 test in 0.002s 2022-03-17T20:49:15.5406454Z 2022-03-17T20:49:15.5406528Z OK (skipped=1) 2022-03-17T20:49:15.5406976Z 2022-03-17T20:49:15.5407101Z Generating XML reports... 2022-03-17T20:49:15.5450074Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204915.xml 2022-03-17T20:49:16.6867498Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:16.6882153Z 2022-03-17T20:49:16.6882675Z Running tests... 2022-03-17T20:49:16.6883091Z ---------------------------------------------------------------------- 2022-03-17T20:49:16.6905217Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:16.6905492Z 2022-03-17T20:49:16.6906173Z ---------------------------------------------------------------------- 2022-03-17T20:49:16.6906459Z Ran 1 test in 0.002s 2022-03-17T20:49:16.6906626Z 2022-03-17T20:49:16.6906738Z OK (skipped=1) 2022-03-17T20:49:16.6906897Z 2022-03-17T20:49:16.6907004Z Generating XML reports... 2022-03-17T20:49:16.6950818Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204916.xml 2022-03-17T20:49:17.8298904Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:17.8314047Z 2022-03-17T20:49:17.8314273Z Running tests... 2022-03-17T20:49:17.8314718Z ---------------------------------------------------------------------- 2022-03-17T20:49:17.8335406Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:17.8335800Z 2022-03-17T20:49:17.8336245Z ---------------------------------------------------------------------- 2022-03-17T20:49:17.8336582Z Ran 1 test in 0.002s 2022-03-17T20:49:17.8336753Z 2022-03-17T20:49:17.8336884Z OK (skipped=1) 2022-03-17T20:49:17.8337030Z 2022-03-17T20:49:17.8337159Z Generating XML reports... 2022-03-17T20:49:17.8380652Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204917.xml 2022-03-17T20:49:18.9801430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:18.9818256Z 2022-03-17T20:49:18.9818680Z Running tests... 2022-03-17T20:49:18.9819176Z ---------------------------------------------------------------------- 2022-03-17T20:49:18.9840833Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:18.9841090Z 2022-03-17T20:49:18.9841976Z ---------------------------------------------------------------------- 2022-03-17T20:49:18.9842605Z Ran 1 test in 0.002s 2022-03-17T20:49:18.9842916Z 2022-03-17T20:49:18.9843123Z OK (skipped=1) 2022-03-17T20:49:18.9843456Z 2022-03-17T20:49:18.9843646Z Generating XML reports... 2022-03-17T20:49:18.9889266Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204918.xml 2022-03-17T20:49:20.1209128Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:20.1224089Z 2022-03-17T20:49:20.1224544Z Running tests... 2022-03-17T20:49:20.1225060Z ---------------------------------------------------------------------- 2022-03-17T20:49:21.6776184Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:49:21.7199478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19254 2022-03-17T20:49:21.7327687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19255 2022-03-17T20:49:22.8043741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:22.8360650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:22.8361425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:22.8448206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:22.8455012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:22.9375323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:25.0410495Z ok (4.918s) 2022-03-17T20:49:25.0410761Z 2022-03-17T20:49:25.0411158Z ---------------------------------------------------------------------- 2022-03-17T20:49:25.0411513Z Ran 1 test in 4.919s 2022-03-17T20:49:25.0411647Z 2022-03-17T20:49:25.0411752Z OK 2022-03-17T20:49:25.0411887Z 2022-03-17T20:49:25.0412026Z Generating XML reports... 2022-03-17T20:49:25.0457457Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204920.xml 2022-03-17T20:49:26.3377124Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:26.3393557Z 2022-03-17T20:49:26.3393976Z Running tests... 2022-03-17T20:49:26.3394472Z ---------------------------------------------------------------------- 2022-03-17T20:49:26.3417223Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:26.3417680Z 2022-03-17T20:49:26.3417976Z ---------------------------------------------------------------------- 2022-03-17T20:49:26.3418356Z Ran 1 test in 0.002s 2022-03-17T20:49:26.3418507Z 2022-03-17T20:49:26.3418617Z OK (skipped=1) 2022-03-17T20:49:26.3418777Z 2022-03-17T20:49:26.3418900Z Generating XML reports... 2022-03-17T20:49:26.3464224Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204926.xml 2022-03-17T20:49:27.4639312Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:27.4654118Z 2022-03-17T20:49:27.4654338Z Running tests... 2022-03-17T20:49:27.4654805Z ---------------------------------------------------------------------- 2022-03-17T20:49:27.4676976Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:27.4677430Z 2022-03-17T20:49:27.4677856Z ---------------------------------------------------------------------- 2022-03-17T20:49:27.4678218Z Ran 1 test in 0.002s 2022-03-17T20:49:27.4678408Z 2022-03-17T20:49:27.4678521Z OK (skipped=1) 2022-03-17T20:49:27.4678625Z 2022-03-17T20:49:27.4678757Z Generating XML reports... 2022-03-17T20:49:27.4722793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204927.xml 2022-03-17T20:49:28.5637641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:28.5653127Z 2022-03-17T20:49:28.5653335Z Running tests... 2022-03-17T20:49:28.5653839Z ---------------------------------------------------------------------- 2022-03-17T20:49:30.1205598Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:49:30.1638025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19450 2022-03-17T20:49:30.1766595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19451 2022-03-17T20:49:31.2545154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:31.2594356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:31.2595159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:31.2646891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:31.2653452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:31.3609506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:33.4849434Z ok (4.919s) 2022-03-17T20:49:33.4849968Z 2022-03-17T20:49:33.4850611Z ---------------------------------------------------------------------- 2022-03-17T20:49:33.4851060Z Ran 1 test in 4.920s 2022-03-17T20:49:33.4851198Z 2022-03-17T20:49:33.4851288Z OK 2022-03-17T20:49:33.4851423Z 2022-03-17T20:49:33.4851569Z Generating XML reports... 2022-03-17T20:49:33.4897256Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204928.xml 2022-03-17T20:49:34.7880255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:34.7895141Z 2022-03-17T20:49:34.7895390Z Running tests... 2022-03-17T20:49:34.7896090Z ---------------------------------------------------------------------- 2022-03-17T20:49:36.3218616Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:49:36.3644004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19576 2022-03-17T20:49:36.3769493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19577 2022-03-17T20:49:37.4316433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:37.4607734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:37.4608574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:37.4620123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:37.4627241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:37.5622367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:39.7867508Z ok (4.997s) 2022-03-17T20:49:39.7867759Z 2022-03-17T20:49:39.7868366Z ---------------------------------------------------------------------- 2022-03-17T20:49:39.7868701Z Ran 1 test in 4.997s 2022-03-17T20:49:39.7868866Z 2022-03-17T20:49:39.7868960Z OK 2022-03-17T20:49:39.7869077Z 2022-03-17T20:49:39.7869215Z Generating XML reports... 2022-03-17T20:49:39.7913360Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204934.xml 2022-03-17T20:49:41.0847768Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:41.0863717Z 2022-03-17T20:49:41.0864158Z Running tests... 2022-03-17T20:49:41.0864668Z ---------------------------------------------------------------------- 2022-03-17T20:49:41.0886769Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:41.0887045Z 2022-03-17T20:49:41.0887350Z ---------------------------------------------------------------------- 2022-03-17T20:49:41.0887665Z Ran 1 test in 0.002s 2022-03-17T20:49:41.0887837Z 2022-03-17T20:49:41.0887958Z OK (skipped=1) 2022-03-17T20:49:41.0888108Z 2022-03-17T20:49:41.0888212Z Generating XML reports... 2022-03-17T20:49:41.0932582Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204941.xml 2022-03-17T20:49:42.2402222Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:42.2417579Z 2022-03-17T20:49:42.2418065Z Running tests... 2022-03-17T20:49:42.2418489Z ---------------------------------------------------------------------- 2022-03-17T20:49:42.2439006Z test_scatter (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:42.2439295Z 2022-03-17T20:49:42.2439891Z ---------------------------------------------------------------------- 2022-03-17T20:49:42.2440563Z Ran 1 test in 0.002s 2022-03-17T20:49:42.2440812Z 2022-03-17T20:49:42.2440926Z OK (skipped=1) 2022-03-17T20:49:42.2441084Z 2022-03-17T20:49:42.2441214Z Generating XML reports... 2022-03-17T20:49:42.2485164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204942.xml 2022-03-17T20:49:43.3861347Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:43.3876476Z 2022-03-17T20:49:43.3876888Z Running tests... 2022-03-17T20:49:43.3877361Z ---------------------------------------------------------------------- 2022-03-17T20:49:43.3904721Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:49:43.3905412Z 2022-03-17T20:49:43.3905746Z ---------------------------------------------------------------------- 2022-03-17T20:49:43.3906118Z Ran 1 test in 0.003s 2022-03-17T20:49:43.3906297Z 2022-03-17T20:49:43.3906630Z OK (skipped=1) 2022-03-17T20:49:43.3906799Z 2022-03-17T20:49:43.3906890Z Generating XML reports... 2022-03-17T20:49:43.3950571Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204943.xml 2022-03-17T20:49:44.4851902Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:44.4868315Z 2022-03-17T20:49:44.4868544Z Running tests... 2022-03-17T20:49:44.4868993Z ---------------------------------------------------------------------- 2022-03-17T20:49:44.4889113Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:49:44.4889497Z 2022-03-17T20:49:44.4889977Z ---------------------------------------------------------------------- 2022-03-17T20:49:44.4890367Z Ran 1 test in 0.002s 2022-03-17T20:49:44.4890503Z 2022-03-17T20:49:44.4890615Z OK (skipped=1) 2022-03-17T20:49:44.4890775Z 2022-03-17T20:49:44.4890896Z Generating XML reports... 2022-03-17T20:49:44.4935504Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204944.xml 2022-03-17T20:49:45.6432001Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:45.6447805Z 2022-03-17T20:49:45.6448191Z Running tests... 2022-03-17T20:49:45.6448690Z ---------------------------------------------------------------------- 2022-03-17T20:49:47.1825977Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:49:47.2248378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19842 2022-03-17T20:49:47.2375639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19843 2022-03-17T20:49:48.2805029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:48.3040714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:48.3041718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:48.3108790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:48.3115893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:48.4057080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:49:52.3492330Z ok (6.704s) 2022-03-17T20:49:52.3492677Z 2022-03-17T20:49:52.3493179Z ---------------------------------------------------------------------- 2022-03-17T20:49:52.3493519Z Ran 1 test in 6.704s 2022-03-17T20:49:52.3493666Z 2022-03-17T20:49:52.3493758Z OK 2022-03-17T20:49:52.3493895Z 2022-03-17T20:49:52.3494029Z Generating XML reports... 2022-03-17T20:49:52.3537906Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204945.xml 2022-03-17T20:49:53.6494840Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:49:53.6510872Z 2022-03-17T20:49:53.6511409Z Running tests... 2022-03-17T20:49:53.6512211Z ---------------------------------------------------------------------- 2022-03-17T20:49:55.2096558Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:49:55.2519247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19970 2022-03-17T20:49:55.2646594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19971 2022-03-17T20:49:56.3146330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:49:56.3376795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:49:56.3377787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:56.3450281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:49:56.3457527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:49:56.4392591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:00.3763330Z ok (6.725s) 2022-03-17T20:50:00.3763564Z 2022-03-17T20:50:00.3763974Z ---------------------------------------------------------------------- 2022-03-17T20:50:00.3764291Z Ran 1 test in 6.725s 2022-03-17T20:50:00.3764427Z 2022-03-17T20:50:00.3764521Z OK 2022-03-17T20:50:00.3764657Z 2022-03-17T20:50:00.3764794Z Generating XML reports... 2022-03-17T20:50:00.3808537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204953.xml 2022-03-17T20:50:01.6762213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:01.6778164Z 2022-03-17T20:50:01.6778450Z Running tests... 2022-03-17T20:50:01.6778879Z ---------------------------------------------------------------------- 2022-03-17T20:50:01.6799513Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:01.6799785Z 2022-03-17T20:50:01.6800087Z ---------------------------------------------------------------------- 2022-03-17T20:50:01.6800406Z Ran 1 test in 0.002s 2022-03-17T20:50:01.6800549Z 2022-03-17T20:50:01.6800693Z OK (skipped=1) 2022-03-17T20:50:01.6800854Z 2022-03-17T20:50:01.6800974Z Generating XML reports... 2022-03-17T20:50:01.6846704Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205001.xml 2022-03-17T20:50:02.8352570Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:02.8368139Z 2022-03-17T20:50:02.8368281Z Running tests... 2022-03-17T20:50:02.8368979Z ---------------------------------------------------------------------- 2022-03-17T20:50:02.8389302Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:02.8389633Z 2022-03-17T20:50:02.8389981Z ---------------------------------------------------------------------- 2022-03-17T20:50:02.8390358Z Ran 1 test in 0.002s 2022-03-17T20:50:02.8390652Z 2022-03-17T20:50:02.8390848Z OK (skipped=1) 2022-03-17T20:50:02.8390964Z 2022-03-17T20:50:02.8391093Z Generating XML reports... 2022-03-17T20:50:02.8436229Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205002.xml 2022-03-17T20:50:03.9761470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:03.9776795Z 2022-03-17T20:50:03.9777110Z Running tests... 2022-03-17T20:50:03.9777581Z ---------------------------------------------------------------------- 2022-03-17T20:50:03.9803360Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:50:03.9803620Z 2022-03-17T20:50:03.9803939Z ---------------------------------------------------------------------- 2022-03-17T20:50:03.9804623Z Ran 1 test in 0.003s 2022-03-17T20:50:03.9804769Z 2022-03-17T20:50:03.9804855Z OK (skipped=1) 2022-03-17T20:50:03.9805011Z 2022-03-17T20:50:03.9805138Z Generating XML reports... 2022-03-17T20:50:03.9848149Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205003.xml 2022-03-17T20:50:05.1252259Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:05.1268360Z 2022-03-17T20:50:05.1268617Z Running tests... 2022-03-17T20:50:05.1269083Z ---------------------------------------------------------------------- 2022-03-17T20:50:05.1289056Z test_send_recv (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:05.1289318Z 2022-03-17T20:50:05.1289623Z ---------------------------------------------------------------------- 2022-03-17T20:50:05.1289939Z Ran 1 test in 0.002s 2022-03-17T20:50:05.1290114Z 2022-03-17T20:50:05.1290225Z OK (skipped=1) 2022-03-17T20:50:05.1290377Z 2022-03-17T20:50:05.1290509Z Generating XML reports... 2022-03-17T20:50:05.1334862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205005.xml 2022-03-17T20:50:06.2804060Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:06.2819913Z 2022-03-17T20:50:06.2820216Z Running tests... 2022-03-17T20:50:06.2820946Z ---------------------------------------------------------------------- 2022-03-17T20:50:06.2840518Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:06.2840911Z 2022-03-17T20:50:06.2841407Z ---------------------------------------------------------------------- 2022-03-17T20:50:06.2841728Z Ran 1 test in 0.002s 2022-03-17T20:50:06.2841893Z 2022-03-17T20:50:06.2842002Z OK (skipped=1) 2022-03-17T20:50:06.2842159Z 2022-03-17T20:50:06.2842263Z Generating XML reports... 2022-03-17T20:50:06.2886642Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205006.xml 2022-03-17T20:50:07.4275195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:07.4291069Z 2022-03-17T20:50:07.4291404Z Running tests... 2022-03-17T20:50:07.4291830Z ---------------------------------------------------------------------- 2022-03-17T20:50:07.4312459Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:07.4312968Z 2022-03-17T20:50:07.4313292Z ---------------------------------------------------------------------- 2022-03-17T20:50:07.4313755Z Ran 1 test in 0.002s 2022-03-17T20:50:07.4313992Z 2022-03-17T20:50:07.4314110Z OK (skipped=1) 2022-03-17T20:50:07.4314268Z 2022-03-17T20:50:07.4314375Z Generating XML reports... 2022-03-17T20:50:07.4358489Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205007.xml 2022-03-17T20:50:08.5799434Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:08.5814892Z 2022-03-17T20:50:08.5815179Z Running tests... 2022-03-17T20:50:08.5815620Z ---------------------------------------------------------------------- 2022-03-17T20:50:08.5837266Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:08.5837598Z 2022-03-17T20:50:08.5837838Z ---------------------------------------------------------------------- 2022-03-17T20:50:08.5838171Z Ran 1 test in 0.002s 2022-03-17T20:50:08.5838341Z 2022-03-17T20:50:08.5838431Z OK (skipped=1) 2022-03-17T20:50:08.5838585Z 2022-03-17T20:50:08.5838723Z Generating XML reports... 2022-03-17T20:50:08.5883360Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205008.xml 2022-03-17T20:50:09.7274371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:09.7289915Z 2022-03-17T20:50:09.7290185Z Running tests... 2022-03-17T20:50:09.7290636Z ---------------------------------------------------------------------- 2022-03-17T20:50:09.7310726Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:09.7310997Z 2022-03-17T20:50:09.7311290Z ---------------------------------------------------------------------- 2022-03-17T20:50:09.7311617Z Ran 1 test in 0.002s 2022-03-17T20:50:09.7311784Z 2022-03-17T20:50:09.7311858Z OK (skipped=1) 2022-03-17T20:50:09.7311985Z 2022-03-17T20:50:09.7312112Z Generating XML reports... 2022-03-17T20:50:09.7357016Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205009.xml 2022-03-17T20:50:10.8758656Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:10.8773389Z 2022-03-17T20:50:10.8773689Z Running tests... 2022-03-17T20:50:10.8774152Z ---------------------------------------------------------------------- 2022-03-17T20:50:12.4147485Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:50:12.4568410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20378 2022-03-17T20:50:12.4693964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20379 2022-03-17T20:50:13.5348199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:13.5709400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:13.5710238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:13.5753172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:13.5759888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:13.6725324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:15.2767184Z ok (4.399s) 2022-03-17T20:50:15.2770529Z 2022-03-17T20:50:15.2771093Z ---------------------------------------------------------------------- 2022-03-17T20:50:15.2771508Z Ran 1 test in 4.399s 2022-03-17T20:50:15.2771685Z 2022-03-17T20:50:15.2771711Z OK 2022-03-17T20:50:15.2771847Z 2022-03-17T20:50:15.2771988Z Generating XML reports... 2022-03-17T20:50:15.2817311Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205010.xml 2022-03-17T20:50:16.5737443Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:16.5752603Z 2022-03-17T20:50:16.5753082Z Running tests... 2022-03-17T20:50:16.5753588Z ---------------------------------------------------------------------- 2022-03-17T20:50:18.1157361Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:50:18.1592978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20500 2022-03-17T20:50:18.1722171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20501 2022-03-17T20:50:19.2461236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:19.2775059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:19.2776150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:19.2866453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:19.2873196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:19.3790253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:21.4805762Z ok (4.905s) 2022-03-17T20:50:21.4806091Z 2022-03-17T20:50:21.4806694Z ---------------------------------------------------------------------- 2022-03-17T20:50:21.4807089Z Ran 1 test in 4.905s 2022-03-17T20:50:21.4807277Z 2022-03-17T20:50:21.4807339Z OK 2022-03-17T20:50:21.4807465Z 2022-03-17T20:50:21.4807598Z Generating XML reports... 2022-03-17T20:50:21.4860845Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205016.xml 2022-03-17T20:50:22.7862093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:22.7876181Z 2022-03-17T20:50:22.7876581Z Running tests... 2022-03-17T20:50:22.7877501Z ---------------------------------------------------------------------- 2022-03-17T20:50:24.3130291Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:50:24.3551874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20626 2022-03-17T20:50:24.3679551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20627 2022-03-17T20:50:25.4451781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:25.4720305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:25.4721752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:25.4755579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:25.4762309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:25.5736652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:27.6760950Z ok (4.888s) 2022-03-17T20:50:27.6761177Z 2022-03-17T20:50:27.6761612Z ---------------------------------------------------------------------- 2022-03-17T20:50:27.6761913Z Ran 1 test in 4.888s 2022-03-17T20:50:27.6762095Z 2022-03-17T20:50:27.6762187Z OK 2022-03-17T20:50:27.6762300Z 2022-03-17T20:50:27.6762433Z Generating XML reports... 2022-03-17T20:50:27.6808278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205022.xml 2022-03-17T20:50:28.9752365Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:28.9767138Z 2022-03-17T20:50:28.9767572Z Running tests... 2022-03-17T20:50:28.9768064Z ---------------------------------------------------------------------- 2022-03-17T20:50:28.9788305Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:28.9788650Z 2022-03-17T20:50:28.9788959Z ---------------------------------------------------------------------- 2022-03-17T20:50:28.9789290Z Ran 1 test in 0.002s 2022-03-17T20:50:28.9789455Z 2022-03-17T20:50:28.9789571Z OK (skipped=1) 2022-03-17T20:50:28.9789624Z 2022-03-17T20:50:28.9789751Z Generating XML reports... 2022-03-17T20:50:28.9833491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205028.xml 2022-03-17T20:50:30.1332164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:30.1348245Z 2022-03-17T20:50:30.1348396Z Running tests... 2022-03-17T20:50:30.1349110Z ---------------------------------------------------------------------- 2022-03-17T20:50:30.1368274Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:30.1368581Z 2022-03-17T20:50:30.1368851Z ---------------------------------------------------------------------- 2022-03-17T20:50:30.1369466Z Ran 1 test in 0.002s 2022-03-17T20:50:30.1369613Z 2022-03-17T20:50:30.1369725Z OK (skipped=1) 2022-03-17T20:50:30.1369881Z 2022-03-17T20:50:30.1370009Z Generating XML reports... 2022-03-17T20:50:30.1415192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205030.xml 2022-03-17T20:50:31.2832117Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:31.2847423Z 2022-03-17T20:50:31.2847848Z Running tests... 2022-03-17T20:50:31.2848363Z ---------------------------------------------------------------------- 2022-03-17T20:50:31.2867941Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:31.2868242Z 2022-03-17T20:50:31.2868522Z ---------------------------------------------------------------------- 2022-03-17T20:50:31.2868852Z Ran 1 test in 0.002s 2022-03-17T20:50:31.2869023Z 2022-03-17T20:50:31.2869132Z OK (skipped=1) 2022-03-17T20:50:31.2869268Z 2022-03-17T20:50:31.2869463Z Generating XML reports... 2022-03-17T20:50:31.2912937Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205031.xml 2022-03-17T20:50:32.4452625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:32.4468317Z 2022-03-17T20:50:32.4468719Z Running tests... 2022-03-17T20:50:32.4469231Z ---------------------------------------------------------------------- 2022-03-17T20:50:32.4490043Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:32.4490356Z 2022-03-17T20:50:32.4490690Z ---------------------------------------------------------------------- 2022-03-17T20:50:32.4490995Z Ran 1 test in 0.002s 2022-03-17T20:50:32.4491132Z 2022-03-17T20:50:32.4491246Z OK (skipped=1) 2022-03-17T20:50:32.4491370Z 2022-03-17T20:50:32.4491497Z Generating XML reports... 2022-03-17T20:50:32.4536285Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205032.xml 2022-03-17T20:50:33.5527723Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:33.5546103Z 2022-03-17T20:50:33.5546596Z Running tests... 2022-03-17T20:50:33.5547114Z ---------------------------------------------------------------------- 2022-03-17T20:50:33.5565451Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:33.5565713Z 2022-03-17T20:50:33.5565988Z ---------------------------------------------------------------------- 2022-03-17T20:50:33.5566386Z Ran 1 test in 0.002s 2022-03-17T20:50:33.5566571Z 2022-03-17T20:50:33.5566674Z OK (skipped=1) 2022-03-17T20:50:33.5566819Z 2022-03-17T20:50:33.5566932Z Generating XML reports... 2022-03-17T20:50:33.5611459Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205033.xml 2022-03-17T20:50:34.7134907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:34.7150787Z 2022-03-17T20:50:34.7151045Z Running tests... 2022-03-17T20:50:34.7151480Z ---------------------------------------------------------------------- 2022-03-17T20:50:34.7170865Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:34.7171676Z 2022-03-17T20:50:34.7171959Z ---------------------------------------------------------------------- 2022-03-17T20:50:34.7172347Z Ran 1 test in 0.002s 2022-03-17T20:50:34.7172461Z 2022-03-17T20:50:34.7172567Z OK (skipped=1) 2022-03-17T20:50:34.7172705Z 2022-03-17T20:50:34.7172853Z Generating XML reports... 2022-03-17T20:50:34.7216994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205034.xml 2022-03-17T20:50:35.8703974Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:35.8719813Z 2022-03-17T20:50:35.8720070Z Running tests... 2022-03-17T20:50:35.8720527Z ---------------------------------------------------------------------- 2022-03-17T20:50:37.4444473Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:50:37.4878192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20962 2022-03-17T20:50:37.5002551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20963 2022-03-17T20:50:38.6002632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:38.6211351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:38.6212135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:38.6306218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:38.6312933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:38.7227227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:41.7104186Z ok (5.838s) 2022-03-17T20:50:41.7104411Z 2022-03-17T20:50:41.7104854Z ---------------------------------------------------------------------- 2022-03-17T20:50:41.7105163Z Ran 1 test in 5.838s 2022-03-17T20:50:41.7105321Z 2022-03-17T20:50:41.7105392Z OK 2022-03-17T20:50:41.7105528Z 2022-03-17T20:50:41.7105663Z Generating XML reports... 2022-03-17T20:50:41.7149745Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205035.xml 2022-03-17T20:50:42.9947959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:42.9962752Z 2022-03-17T20:50:42.9963137Z Running tests... 2022-03-17T20:50:42.9963614Z ---------------------------------------------------------------------- 2022-03-17T20:50:42.9986420Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:50:42.9986707Z 2022-03-17T20:50:42.9987057Z ---------------------------------------------------------------------- 2022-03-17T20:50:42.9987380Z Ran 1 test in 0.002s 2022-03-17T20:50:42.9987543Z 2022-03-17T20:50:42.9987657Z OK (skipped=1) 2022-03-17T20:50:42.9987810Z 2022-03-17T20:50:42.9990149Z Generating XML reports... 2022-03-17T20:50:43.0030751Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205042.xml 2022-03-17T20:50:44.1429100Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:44.1444229Z 2022-03-17T20:50:44.1444691Z Running tests... 2022-03-17T20:50:44.1445198Z ---------------------------------------------------------------------- 2022-03-17T20:50:45.6869518Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:50:45.7307637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21124 2022-03-17T20:50:45.7435630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21125 2022-03-17T20:50:46.8135503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:46.8245158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:46.8245979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:46.8338170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:46.8345191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:46.9259838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:49.5525720Z ok (5.408s) 2022-03-17T20:50:49.5526206Z 2022-03-17T20:50:49.5526701Z ---------------------------------------------------------------------- 2022-03-17T20:50:49.5527046Z Ran 1 test in 5.408s 2022-03-17T20:50:49.5527215Z 2022-03-17T20:50:49.5527288Z OK 2022-03-17T20:50:49.5527424Z 2022-03-17T20:50:49.5527560Z Generating XML reports... 2022-03-17T20:50:49.5581196Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205044.xml 2022-03-17T20:50:50.8329315Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:50.8344103Z 2022-03-17T20:50:50.8344520Z Running tests... 2022-03-17T20:50:50.8345036Z ---------------------------------------------------------------------- 2022-03-17T20:50:52.3604728Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:50:52.4023727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21247 2022-03-17T20:50:52.4151840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21248 2022-03-17T20:50:53.4496984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:50:53.4660645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:50:53.4661722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:53.4699171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:50:53.4706762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:50:53.5675844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:50:56.1684298Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:50:56.1825348Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:50:56.5247531Z ok (5.690s) 2022-03-17T20:50:56.5247880Z 2022-03-17T20:50:56.5248513Z ---------------------------------------------------------------------- 2022-03-17T20:50:56.5249121Z Ran 1 test in 5.690s 2022-03-17T20:50:56.5249403Z 2022-03-17T20:50:56.5249560Z OK 2022-03-17T20:50:56.5249794Z 2022-03-17T20:50:56.5250025Z Generating XML reports... 2022-03-17T20:50:56.5295245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205050.xml 2022-03-17T20:50:57.8211704Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:50:57.8227317Z 2022-03-17T20:50:57.8227558Z Running tests... 2022-03-17T20:50:57.8227977Z ---------------------------------------------------------------------- 2022-03-17T20:50:59.3698254Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:50:59.4121924Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21374 2022-03-17T20:50:59.4248231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21375 2022-03-17T20:51:00.5226676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:00.5309662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:00.5311037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:00.5328074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:00.5334320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:00.6323011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:00.6452661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:51:00.6453305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:51:00.6453956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:51:00.6454640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:51:00.6456303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:51:00.6457116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:51:00.6457692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:51:00.6458462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:51:07.6400013Z ok (9.817s) 2022-03-17T20:51:07.6400274Z 2022-03-17T20:51:07.6400688Z ---------------------------------------------------------------------- 2022-03-17T20:51:07.6401028Z Ran 1 test in 9.817s 2022-03-17T20:51:07.6401193Z 2022-03-17T20:51:07.6401266Z OK 2022-03-17T20:51:07.6401400Z 2022-03-17T20:51:07.6401533Z Generating XML reports... 2022-03-17T20:51:07.6453493Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205057.xml 2022-03-17T20:51:08.9441255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn 2022-03-17T20:51:08.9456195Z 2022-03-17T20:51:08.9456317Z Running tests... 2022-03-17T20:51:08.9457070Z ---------------------------------------------------------------------- 2022-03-17T20:51:10.4803217Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:51:10.5240428Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21504 2022-03-17T20:51:10.5371155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21505 2022-03-17T20:51:11.5975059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:11.6173936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:11.6174745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:11.6177536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:11.6184399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:11.7185810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:11.7397000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:51:11.7397519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:51:11.7398195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:51:11.7398887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:51:11.7400997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:51:11.7401496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:51:11.7402177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:51:11.7402863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:51:18.6536390Z ok (9.708s) 2022-03-17T20:51:18.6536787Z 2022-03-17T20:51:18.6537483Z ---------------------------------------------------------------------- 2022-03-17T20:51:18.6537907Z Ran 1 test in 9.708s 2022-03-17T20:51:18.6538016Z 2022-03-17T20:51:18.6538144Z OK 2022-03-17T20:51:18.6538286Z 2022-03-17T20:51:18.6538419Z Generating XML reports... 2022-03-17T20:51:18.6582802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205108.xml 2022-03-17T20:51:19.0065828Z Running distributed tests for the gloo backend with env init_method 2022-03-17T20:51:19.0066738Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 20:51:19.006319] 2022-03-17T20:51:20.0584849Z 2022-03-17T20:51:20.0624730Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_get_bucket_sizes>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_params_and_buffers>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-03-17T20:51:20.0658010Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0658501Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0658929Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0659325Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0659772Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0660651Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0661306Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0661811Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0662304Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0662850Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0663403Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0663906Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0664434Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0664946Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0665407Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0665891Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0666337Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0666767Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0667193Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0667617Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0668091Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0668541Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0668948Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0669341Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0669751Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0670174Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0670568Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0670989Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0671402Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0671765Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0672155Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0672535Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0672923Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0673293Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0673699Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0674121Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0674518Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0674944Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0675363Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0675896Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0676338Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0676743Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0677167Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0677576Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0678006Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0678488Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0678913Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0679347Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0679743Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0680151Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0680572Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0680975Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0681376Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0681816Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0682227Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0682628Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0683000Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0683398Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0683775Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0684156Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0684528Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0684892Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0685298Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0685681Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0686070Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0686450Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0686812Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0687204Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0687578Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0687970Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0688361Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0688745Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0689116Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0689471Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0689857Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0690232Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0690630Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0691020Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0691389Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0691793Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0692203Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0692703Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0693142Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0693570Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0694021Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0694444Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0694884Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0695315Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0695780Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0696226Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0696651Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0697111Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0697568Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0697998Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0698444Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0698848Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0699239Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0699622Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0699966Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0700823Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0701244Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0701641Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0702026Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0702389Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0702789Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0703173Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0703566Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0703959Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0704344Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0704767Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0705164Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0705576Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0705971Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0706379Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0706787Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0707177Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0707562Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0707912Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0708302Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0708688Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0709060Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0709450Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0709878Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0710490Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0710936Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0711325Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0711736Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0712146Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0712588Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0713118Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0713544Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0713961Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0714378Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0714790Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0715161Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0715528Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0715926Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0716309Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0716720Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0717156Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0717578Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0717992Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0718423Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0718923Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0719476Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0720064Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0720654Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0721248Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0721831Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0722434Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0723027Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0723600Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0724138Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0724612Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0725055Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0725439Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0725817Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0726289Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0726665Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0727079Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0727517Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0727953Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0728413Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0728806Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0729247Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0729663Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0730074Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0730506Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0730900Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0731313Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0731734Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0732141Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0732557Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0732951Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0733372Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0733789Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0734170Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0734588Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0735027Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0735452Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0735830Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0736207Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0736619Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0737013Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0737393Z test_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0737739Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0738106Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0738483Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0738841Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0739215Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0739584Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0739966Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0740826Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0741185Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0741564Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0741941Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0742331Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0742707Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0743037Z test_isend (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0743414Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0743797Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0744301Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0744733Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0745180Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0745592Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0745991Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0746421Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0746885Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0747317Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0747729Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0748122Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0748534Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0748924Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0749315Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0749711Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0750153Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0750640Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0751091Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0751537Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0751989Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0752426Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0752854Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0753252Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0753666Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0754077Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0754448Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0754846Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0755232Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0755618Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0755995Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0756364Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0756753Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0757102Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0757469Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0757841Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0758200Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0758567Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0758920Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0759303Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0759670Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0760039Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0760402Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0760758Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0761193Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0761556Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0761951Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0762329Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0762689Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0763059Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0763414Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0763829Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0764320Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0764732Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0765124Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0765506Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0765927Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0766340Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0766714Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0767122Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0767536Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0767948Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0768332Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0768738Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0769133Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0769495Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0769914Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0770339Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:20.0770775Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T20:51:21.1102058Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:51:21.1119652Z 2022-03-17T20:51:21.1119786Z Running tests... 2022-03-17T20:51:21.1120462Z ---------------------------------------------------------------------- 2022-03-17T20:51:22.6564325Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:51:22.6994418Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21669 2022-03-17T20:51:22.7125486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21670 2022-03-17T20:51:23.7784031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:23.7784559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:23.7785356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:23.7786041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:23.7793450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:23.7794642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:25.1040123Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:51:25.1041121Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:51:25.1155677Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:51:25.1156544Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:51:26.1166413Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:51:26.1167013Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:51:26.1168259Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:51:26.1169213Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:51:26.1326023Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:51:26.1326606Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:51:26.1327456Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:51:26.1328348Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:51:26.1475529Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:51:26.1476110Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T20:51:26.1476937Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:51:26.1477844Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T20:51:26.5215340Z ok (5.409s) 2022-03-17T20:51:26.5215523Z 2022-03-17T20:51:26.5215937Z ---------------------------------------------------------------------- 2022-03-17T20:51:26.5216257Z Ran 1 test in 5.410s 2022-03-17T20:51:26.5216428Z 2022-03-17T20:51:26.5216535Z OK 2022-03-17T20:51:26.5216639Z 2022-03-17T20:51:26.5216768Z Generating XML reports... 2022-03-17T20:51:26.5259384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205121.xml 2022-03-17T20:51:27.8207979Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:51:27.8222676Z 2022-03-17T20:51:27.8222830Z Running tests... 2022-03-17T20:51:27.8223712Z ---------------------------------------------------------------------- 2022-03-17T20:51:27.8271376Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip (0.005s) 2022-03-17T20:51:27.8271735Z 2022-03-17T20:51:27.8272032Z ---------------------------------------------------------------------- 2022-03-17T20:51:27.8272342Z Ran 1 test in 0.005s 2022-03-17T20:51:27.8272503Z 2022-03-17T20:51:27.8272621Z OK (skipped=1) 2022-03-17T20:51:27.8272776Z 2022-03-17T20:51:27.8272870Z Generating XML reports... 2022-03-17T20:51:27.8321771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205127.xml 2022-03-17T20:51:28.9743966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:51:28.9759129Z 2022-03-17T20:51:28.9759425Z Running tests... 2022-03-17T20:51:28.9759866Z ---------------------------------------------------------------------- 2022-03-17T20:51:30.5237268Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:51:30.5667144Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21822 2022-03-17T20:51:30.5797799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21823 2022-03-17T20:51:31.7014354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:31.7015101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:31.7015914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:31.7016586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:31.7123708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:31.8027362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:31.9847653Z ok (3.008s) 2022-03-17T20:51:31.9847832Z 2022-03-17T20:51:31.9848224Z ---------------------------------------------------------------------- 2022-03-17T20:51:31.9848565Z Ran 1 test in 3.009s 2022-03-17T20:51:31.9848731Z 2022-03-17T20:51:31.9848803Z OK 2022-03-17T20:51:31.9848946Z 2022-03-17T20:51:31.9849080Z Generating XML reports... 2022-03-17T20:51:31.9893631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205128.xml 2022-03-17T20:51:33.2640535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:51:33.2654502Z 2022-03-17T20:51:33.2654699Z Running tests... 2022-03-17T20:51:33.2655433Z ---------------------------------------------------------------------- 2022-03-17T20:51:34.8028649Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:51:34.8454909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21937 2022-03-17T20:51:34.8580595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21938 2022-03-17T20:51:35.9736261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:35.9736823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:35.9737568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:35.9738257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:35.9744582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:35.9745271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:37.6067123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.6067668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.6259039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.6259513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.6337200Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:51:37.6338240Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:51:37.6339635Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:51:37.6340662Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:51:37.6586726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.6587216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.7971541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.7972025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.8161016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.8161527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.8404674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.8405156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.8594269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.8594788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.9900377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:37.9901109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.1190127Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:51:38.1190714Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:51:38.1191363Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T20:51:38.1191898Z "You've set static_graph to be True, no need to set it again." 2022-03-17T20:51:38.1358437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.1358962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.2819542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.2820075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.4282558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.4283066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.5670607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.5671142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.5858794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.5859308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.7167969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.7168518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.8576714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.8577405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.8786362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:38.8786916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.0050578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.0051542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.0254232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.0255420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.1577052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.1577966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.3030196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.3030734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.4460978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.4461946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.5894016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.5894948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:39.9693227Z ok (6.704s) 2022-03-17T20:51:39.9693463Z 2022-03-17T20:51:39.9693854Z ---------------------------------------------------------------------- 2022-03-17T20:51:39.9694177Z Ran 1 test in 6.704s 2022-03-17T20:51:39.9694344Z 2022-03-17T20:51:39.9694460Z OK 2022-03-17T20:51:39.9694598Z 2022-03-17T20:51:39.9694714Z Generating XML reports... 2022-03-17T20:51:39.9736155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205133.xml 2022-03-17T20:51:41.2751267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:51:41.2765351Z 2022-03-17T20:51:41.2765487Z Running tests... 2022-03-17T20:51:41.2766153Z ---------------------------------------------------------------------- 2022-03-17T20:51:42.8133235Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:51:42.8562872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22058 2022-03-17T20:51:42.8692203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22059 2022-03-17T20:51:43.9253122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:43.9253685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:43.9254467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:43.9255160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:43.9361674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:44.0264072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:44.0346983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:44.0347495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:44.0507649Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:51:44.0508749Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:51:44.0509850Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:51:44.0510766Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:51:44.2738203Z ok (2.997s) 2022-03-17T20:51:44.2738733Z 2022-03-17T20:51:44.2739316Z ---------------------------------------------------------------------- 2022-03-17T20:51:44.2739830Z Ran 1 test in 2.997s 2022-03-17T20:51:44.2739973Z 2022-03-17T20:51:44.2740065Z OK 2022-03-17T20:51:44.2740454Z 2022-03-17T20:51:44.2740678Z Generating XML reports... 2022-03-17T20:51:44.2782159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205141.xml 2022-03-17T20:51:45.5562685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:51:45.5577796Z 2022-03-17T20:51:45.5577977Z Running tests... 2022-03-17T20:51:45.5578846Z ---------------------------------------------------------------------- 2022-03-17T20:51:47.0886014Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:51:47.1321805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22177 2022-03-17T20:51:47.1453078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22178 2022-03-17T20:51:48.2202886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:48.2203467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:48.2204268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:48.2204945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:48.2311838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:48.3214430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:48.3295264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:48.3295780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:48.3443248Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:51:48.3444032Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:51:48.3445070Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:51:48.3445835Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:51:48.5501339Z ok (2.992s) 2022-03-17T20:51:48.5501798Z 2022-03-17T20:51:48.5502185Z ---------------------------------------------------------------------- 2022-03-17T20:51:48.5502548Z Ran 1 test in 2.992s 2022-03-17T20:51:48.5502712Z 2022-03-17T20:51:48.5502762Z OK 2022-03-17T20:51:48.5502897Z 2022-03-17T20:51:48.5503027Z Generating XML reports... 2022-03-17T20:51:48.5546166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205145.xml 2022-03-17T20:51:49.8494099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:51:49.8509816Z 2022-03-17T20:51:49.8510062Z Running tests... 2022-03-17T20:51:49.8510800Z ---------------------------------------------------------------------- 2022-03-17T20:51:51.3976340Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:51:51.4407321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22296 2022-03-17T20:51:51.4533961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22297 2022-03-17T20:51:52.5889604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:52.5890127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:52.5890921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:52.5891625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:52.5897933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:51:52.5898704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:54.5920694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.5921262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.6164290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.6164762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.6262681Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:51:54.6263490Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:51:54.6265126Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:51:54.6265859Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:51:54.6488585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.6489054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.6728218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.6728673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.8030883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.8031358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.8269000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:54.8269655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:51:55.3626757Z ok (5.511s) 2022-03-17T20:51:55.3626984Z 2022-03-17T20:51:55.3627350Z ---------------------------------------------------------------------- 2022-03-17T20:51:55.3627700Z Ran 1 test in 5.512s 2022-03-17T20:51:55.3627865Z 2022-03-17T20:51:55.3627957Z OK 2022-03-17T20:51:55.3628045Z 2022-03-17T20:51:55.3628179Z Generating XML reports... 2022-03-17T20:51:55.3670829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205149.xml 2022-03-17T20:51:56.6533730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:51:56.6547550Z 2022-03-17T20:51:56.6548013Z Running tests... 2022-03-17T20:51:56.6548452Z ---------------------------------------------------------------------- 2022-03-17T20:51:58.1792910Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:51:58.2214914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22417 2022-03-17T20:51:58.2341638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22418 2022-03-17T20:51:59.2967454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:51:59.2967988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:51:59.2968765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:59.2969459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:51:59.3077755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:51:59.3982468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:00.6877782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:00.6878301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:00.7037956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:00.7038490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:00.7107415Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:00.7108138Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:00.7109182Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:00.7109944Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:01.0412108Z ok (4.386s) 2022-03-17T20:52:01.0412330Z 2022-03-17T20:52:01.0412731Z ---------------------------------------------------------------------- 2022-03-17T20:52:01.0413066Z Ran 1 test in 4.386s 2022-03-17T20:52:01.0413231Z 2022-03-17T20:52:01.0413324Z OK 2022-03-17T20:52:01.0413521Z 2022-03-17T20:52:01.0413629Z Generating XML reports... 2022-03-17T20:52:01.0455747Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205156.xml 2022-03-17T20:52:02.3504475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:02.3519350Z 2022-03-17T20:52:02.3519602Z Running tests... 2022-03-17T20:52:02.3520056Z ---------------------------------------------------------------------- 2022-03-17T20:52:03.9085145Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:03.9516408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22538 2022-03-17T20:52:03.9647218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22539 2022-03-17T20:52:05.0416266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:05.0416836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:05.0417625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:05.0418383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:05.0524466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:05.1430263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:06.4208590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:06.4209140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:06.4389892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:06.4390380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:06.4457776Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:06.4458576Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:06.4459612Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:06.4460658Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:06.7721458Z ok (4.420s) 2022-03-17T20:52:06.7723310Z 2022-03-17T20:52:06.7723910Z ---------------------------------------------------------------------- 2022-03-17T20:52:06.7724210Z Ran 1 test in 4.420s 2022-03-17T20:52:06.7724386Z 2022-03-17T20:52:06.7724478Z OK 2022-03-17T20:52:06.7724615Z 2022-03-17T20:52:06.7724758Z Generating XML reports... 2022-03-17T20:52:06.7766093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205202.xml 2022-03-17T20:52:08.0814955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:08.0830600Z 2022-03-17T20:52:08.0830931Z Running tests... 2022-03-17T20:52:08.0831372Z ---------------------------------------------------------------------- 2022-03-17T20:52:09.6176648Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:09.6604309Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22659 2022-03-17T20:52:09.6732601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22660 2022-03-17T20:52:10.7665065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:10.7665591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:10.7666403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:10.7667099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:10.7775151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:10.8679507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:12.1465594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:12.1466162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:12.5807967Z ok (4.497s) 2022-03-17T20:52:12.5808319Z 2022-03-17T20:52:12.5808702Z ---------------------------------------------------------------------- 2022-03-17T20:52:12.5809013Z Ran 1 test in 4.498s 2022-03-17T20:52:12.5809178Z 2022-03-17T20:52:12.5809271Z OK 2022-03-17T20:52:12.5809388Z 2022-03-17T20:52:12.5809524Z Generating XML reports... 2022-03-17T20:52:12.5851882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205208.xml 2022-03-17T20:52:13.8813661Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:13.8829286Z 2022-03-17T20:52:13.8829831Z Running tests... 2022-03-17T20:52:13.8830294Z ---------------------------------------------------------------------- 2022-03-17T20:52:15.4285948Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:15.4722053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22780 2022-03-17T20:52:15.4850297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22781 2022-03-17T20:52:16.5897627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:16.5898114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:16.5898886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:16.5899598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:16.5906276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:16.5906696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:18.5743537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:18.5744061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:18.5986977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:18.5987448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:18.6084731Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:18.6085779Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:18.6088057Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:18.6088812Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:18.9936000Z ok (5.110s) 2022-03-17T20:52:18.9936197Z 2022-03-17T20:52:18.9936695Z ---------------------------------------------------------------------- 2022-03-17T20:52:18.9937044Z Ran 1 test in 5.111s 2022-03-17T20:52:18.9937183Z 2022-03-17T20:52:18.9937528Z OK 2022-03-17T20:52:18.9937658Z 2022-03-17T20:52:18.9937730Z Generating XML reports... 2022-03-17T20:52:18.9979290Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205213.xml 2022-03-17T20:52:20.2923828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:20.2937050Z 2022-03-17T20:52:20.2937496Z Running tests... 2022-03-17T20:52:20.2937998Z ---------------------------------------------------------------------- 2022-03-17T20:52:21.8223252Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:21.8649104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22901 2022-03-17T20:52:21.8775388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22902 2022-03-17T20:52:22.9246506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:22.9249752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:22.9250545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:22.9347853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:22.9354798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:23.0264761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:24.6141571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:24.6142075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:24.6340716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:24.6341241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:24.6421373Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:24.6422136Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:24.6423184Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:24.6423959Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:24.9856558Z ok (4.692s) 2022-03-17T20:52:24.9856789Z 2022-03-17T20:52:24.9857179Z ---------------------------------------------------------------------- 2022-03-17T20:52:24.9857734Z Ran 1 test in 4.692s 2022-03-17T20:52:24.9857907Z 2022-03-17T20:52:24.9858002Z OK 2022-03-17T20:52:24.9858141Z 2022-03-17T20:52:24.9858271Z Generating XML reports... 2022-03-17T20:52:24.9902192Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205220.xml 2022-03-17T20:52:26.2858678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:26.2873203Z 2022-03-17T20:52:26.2873752Z Running tests... 2022-03-17T20:52:26.2874240Z ---------------------------------------------------------------------- 2022-03-17T20:52:27.8177073Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:27.8601217Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23022 2022-03-17T20:52:27.8728716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23023 2022-03-17T20:52:28.9689564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:28.9690106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:28.9690849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:28.9691541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:28.9800522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:29.0703678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:30.3617851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.3618318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.3779393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.3779896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:30.3847887Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:30.3848694Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:30.3849745Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T20:52:30.3850513Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T20:52:30.6802458Z ok (4.393s) 2022-03-17T20:52:30.6802669Z 2022-03-17T20:52:30.6803031Z ---------------------------------------------------------------------- 2022-03-17T20:52:30.6803365Z Ran 1 test in 4.393s 2022-03-17T20:52:30.6803536Z 2022-03-17T20:52:30.6803622Z OK 2022-03-17T20:52:30.6803704Z 2022-03-17T20:52:30.6803834Z Generating XML reports... 2022-03-17T20:52:30.6846398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205226.xml 2022-03-17T20:52:31.9772450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:31.9786727Z 2022-03-17T20:52:31.9787033Z Running tests... 2022-03-17T20:52:31.9787471Z ---------------------------------------------------------------------- 2022-03-17T20:52:33.5163374Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:33.5590903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23143 2022-03-17T20:52:33.5722140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23144 2022-03-17T20:52:34.6607007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:34.6608075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:34.6609578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:34.6610948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:34.6615107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:34.6617939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:36.2782509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:36.2783076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:38.6833722Z ok (6.704s) 2022-03-17T20:52:38.6833916Z 2022-03-17T20:52:38.6834306Z ---------------------------------------------------------------------- 2022-03-17T20:52:38.6834664Z Ran 1 test in 6.705s 2022-03-17T20:52:38.6834833Z 2022-03-17T20:52:38.6834909Z OK 2022-03-17T20:52:38.6835046Z 2022-03-17T20:52:38.6835184Z Generating XML reports... 2022-03-17T20:52:38.6878052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205231.xml 2022-03-17T20:52:39.9693531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:39.9707270Z 2022-03-17T20:52:39.9707752Z Running tests... 2022-03-17T20:52:39.9708228Z ---------------------------------------------------------------------- 2022-03-17T20:52:41.4832021Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:41.5253867Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23264 2022-03-17T20:52:41.5380415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23265 2022-03-17T20:52:42.6377688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:42.6378331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:42.6379034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:42.6379711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:42.6487636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:42.7390863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:43.0431246Z ok (3.072s) 2022-03-17T20:52:43.0431450Z 2022-03-17T20:52:43.0431919Z ---------------------------------------------------------------------- 2022-03-17T20:52:43.0432242Z Ran 1 test in 3.072s 2022-03-17T20:52:43.0432416Z 2022-03-17T20:52:43.0432498Z OK 2022-03-17T20:52:43.0432622Z 2022-03-17T20:52:43.0434163Z Generating XML reports... 2022-03-17T20:52:43.0476870Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205239.xml 2022-03-17T20:52:44.3339596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:44.3355656Z 2022-03-17T20:52:44.3356146Z Running tests... 2022-03-17T20:52:44.3357033Z ---------------------------------------------------------------------- 2022-03-17T20:52:45.8967041Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:45.9399951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23379 2022-03-17T20:52:45.9532056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23380 2022-03-17T20:52:47.0479405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:47.0480244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:47.0480995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:47.0481700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:47.0588874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:47.1494319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:48.7186834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:48.7187394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:48.7811282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:48.7811813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:52:49.1623026Z ok (4.826s) 2022-03-17T20:52:49.1623232Z 2022-03-17T20:52:49.1623844Z ---------------------------------------------------------------------- 2022-03-17T20:52:49.1624216Z Ran 1 test in 4.827s 2022-03-17T20:52:49.1624389Z 2022-03-17T20:52:49.1624493Z OK 2022-03-17T20:52:49.1624623Z 2022-03-17T20:52:49.1624757Z Generating XML reports... 2022-03-17T20:52:49.1669416Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205244.xml 2022-03-17T20:52:50.4619859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:50.4634659Z 2022-03-17T20:52:50.4635108Z Running tests... 2022-03-17T20:52:50.4635660Z ---------------------------------------------------------------------- 2022-03-17T20:52:52.0136370Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:52.0562757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23500 2022-03-17T20:52:52.0690366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23501 2022-03-17T20:52:53.1297615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:53.1298440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:53.1299207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:53.1299878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:53.1406365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:52:53.2311659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:52:54.7763096Z ok (4.312s) 2022-03-17T20:52:54.7763363Z 2022-03-17T20:52:54.7763748Z ---------------------------------------------------------------------- 2022-03-17T20:52:54.7764075Z Ran 1 test in 4.313s 2022-03-17T20:52:54.7764246Z 2022-03-17T20:52:54.7764339Z OK 2022-03-17T20:52:54.7764446Z 2022-03-17T20:52:54.7764588Z Generating XML reports... 2022-03-17T20:52:54.7806052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205250.xml 2022-03-17T20:52:56.0890307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:56.0906684Z 2022-03-17T20:52:56.0907144Z Running tests... 2022-03-17T20:52:56.0907676Z ---------------------------------------------------------------------- 2022-03-17T20:52:56.0929319Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:52:56.0929648Z 2022-03-17T20:52:56.0930346Z ---------------------------------------------------------------------- 2022-03-17T20:52:56.0930925Z Ran 1 test in 0.002s 2022-03-17T20:52:56.0931111Z 2022-03-17T20:52:56.0931216Z OK (skipped=1) 2022-03-17T20:52:56.0931376Z 2022-03-17T20:52:56.0931507Z Generating XML reports... 2022-03-17T20:52:56.0977119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205256.xml 2022-03-17T20:52:57.2392591Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:52:57.2406297Z 2022-03-17T20:52:57.2406547Z Running tests... 2022-03-17T20:52:57.2406984Z ---------------------------------------------------------------------- 2022-03-17T20:52:57.2426030Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T20:52:58.7994690Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:52:58.8419872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23652 2022-03-17T20:52:58.8549111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23653 2022-03-17T20:52:59.9404130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:52:59.9404714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:52:59.9405496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:59.9406183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:52:59.9512054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:00.0414202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:00.2596900Z ok (3.019s) 2022-03-17T20:53:00.2597317Z 2022-03-17T20:53:00.2598007Z ---------------------------------------------------------------------- 2022-03-17T20:53:00.2598592Z Ran 1 test in 3.019s 2022-03-17T20:53:00.2598867Z 2022-03-17T20:53:00.2599036Z OK 2022-03-17T20:53:00.2599271Z 2022-03-17T20:53:00.2599500Z Generating XML reports... 2022-03-17T20:53:00.2642799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205257.xml 2022-03-17T20:53:01.5500888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:01.5514750Z 2022-03-17T20:53:01.5515012Z Running tests... 2022-03-17T20:53:01.5515451Z ---------------------------------------------------------------------- 2022-03-17T20:53:01.5539170Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:53:03.0908458Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:03.1337493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23771 2022-03-17T20:53:03.1465490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23772 2022-03-17T20:53:04.2604176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:04.2605133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:04.2606177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:04.2606863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:04.2612278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:04.2613349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:04.2748954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:53:04.2750132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:53:04.4513462Z ok (2.899s) 2022-03-17T20:53:04.4513859Z 2022-03-17T20:53:04.4514598Z ---------------------------------------------------------------------- 2022-03-17T20:53:04.4515007Z Ran 1 test in 2.900s 2022-03-17T20:53:04.4515172Z 2022-03-17T20:53:04.4515246Z OK 2022-03-17T20:53:04.4515381Z 2022-03-17T20:53:04.4515514Z Generating XML reports... 2022-03-17T20:53:04.4558525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205301.xml 2022-03-17T20:53:05.7450377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:05.7465911Z 2022-03-17T20:53:05.7466403Z Running tests... 2022-03-17T20:53:05.7466997Z ---------------------------------------------------------------------- 2022-03-17T20:53:05.7498095Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T20:53:07.2925230Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:07.3355509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23890 2022-03-17T20:53:07.3483009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23891 2022-03-17T20:53:08.4493984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:08.4494532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:08.4495312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:08.4495989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:08.4602305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:08.5507668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:08.5647853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:53:08.5648382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:53:08.7530529Z ok (3.006s) 2022-03-17T20:53:08.7530882Z 2022-03-17T20:53:08.7531483Z ---------------------------------------------------------------------- 2022-03-17T20:53:08.7531832Z Ran 1 test in 3.006s 2022-03-17T20:53:08.7531994Z 2022-03-17T20:53:08.7532087Z OK 2022-03-17T20:53:08.7532230Z 2022-03-17T20:53:08.7532339Z Generating XML reports... 2022-03-17T20:53:08.7574207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205305.xml 2022-03-17T20:53:10.0418660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:10.0433331Z 2022-03-17T20:53:10.0433828Z Running tests... 2022-03-17T20:53:10.0434342Z ---------------------------------------------------------------------- 2022-03-17T20:53:10.0452942Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T20:53:11.5913761Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:11.6343689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24009 2022-03-17T20:53:11.6473798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24010 2022-03-17T20:53:12.7528128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:12.7528757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:12.7529717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:12.7530417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:12.7536567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:12.7537281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:12.9524442Z ok (2.909s) 2022-03-17T20:53:12.9524649Z 2022-03-17T20:53:12.9525059Z ---------------------------------------------------------------------- 2022-03-17T20:53:12.9525992Z Ran 1 test in 2.909s 2022-03-17T20:53:12.9526207Z 2022-03-17T20:53:12.9526307Z OK 2022-03-17T20:53:12.9526457Z 2022-03-17T20:53:12.9526595Z Generating XML reports... 2022-03-17T20:53:12.9569643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205310.xml 2022-03-17T20:53:14.2047883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:14.2062482Z 2022-03-17T20:53:14.2062810Z Running tests... 2022-03-17T20:53:14.2063257Z ---------------------------------------------------------------------- 2022-03-17T20:53:15.7403568Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:15.7828905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24128 2022-03-17T20:53:15.7958162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24129 2022-03-17T20:53:16.8860518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:16.8861094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:16.8861811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:16.8862530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:16.8970124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:16.9871131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:17.2010518Z ok (2.994s) 2022-03-17T20:53:17.2010795Z 2022-03-17T20:53:17.2011541Z ---------------------------------------------------------------------- 2022-03-17T20:53:17.2011887Z Ran 1 test in 2.995s 2022-03-17T20:53:17.2012080Z 2022-03-17T20:53:17.2012166Z OK 2022-03-17T20:53:17.2012315Z 2022-03-17T20:53:17.2012458Z Generating XML reports... 2022-03-17T20:53:17.2055798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205314.xml 2022-03-17T20:53:18.4862823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:18.4877229Z 2022-03-17T20:53:18.4877779Z Running tests... 2022-03-17T20:53:18.4878297Z ---------------------------------------------------------------------- 2022-03-17T20:53:20.0253511Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:20.0680827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24247 2022-03-17T20:53:20.0808094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24248 2022-03-17T20:53:21.1832924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:21.1833422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:21.1834203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:21.1835110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:21.1942875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:21.2845321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:21.4857313Z ok (2.998s) 2022-03-17T20:53:21.4857531Z 2022-03-17T20:53:21.4857961Z ---------------------------------------------------------------------- 2022-03-17T20:53:21.4858287Z Ran 1 test in 2.998s 2022-03-17T20:53:21.4858461Z 2022-03-17T20:53:21.4858533Z OK 2022-03-17T20:53:21.4858637Z 2022-03-17T20:53:21.4858758Z Generating XML reports... 2022-03-17T20:53:21.4901352Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205318.xml 2022-03-17T20:53:22.7745586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:22.7759618Z 2022-03-17T20:53:22.7759908Z Running tests... 2022-03-17T20:53:22.7760512Z ---------------------------------------------------------------------- 2022-03-17T20:53:24.3180065Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:24.3608155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24366 2022-03-17T20:53:24.3735931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24367 2022-03-17T20:53:25.4744564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:25.4745127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:25.4745915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:25.4746562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:25.4752705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:25.4753487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:25.4861063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:53:25.4861575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:53:25.4862264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:53:25.4862953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:53:25.6786465Z ok (2.902s) 2022-03-17T20:53:25.6786665Z 2022-03-17T20:53:25.6787097Z ---------------------------------------------------------------------- 2022-03-17T20:53:25.6787449Z Ran 1 test in 2.903s 2022-03-17T20:53:25.6787579Z 2022-03-17T20:53:25.6787687Z OK 2022-03-17T20:53:25.6787786Z 2022-03-17T20:53:25.6787922Z Generating XML reports... 2022-03-17T20:53:25.6829967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205322.xml 2022-03-17T20:53:26.9661744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:26.9677671Z 2022-03-17T20:53:26.9677908Z Running tests... 2022-03-17T20:53:26.9678617Z ---------------------------------------------------------------------- 2022-03-17T20:53:28.5222838Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:28.5651433Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24491 2022-03-17T20:53:28.5781434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24492 2022-03-17T20:53:29.6811315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:29.6811867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:29.6812669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:29.6813383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:29.6818867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:29.6819353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:29.8829888Z skip (2.915s) 2022-03-17T20:53:29.8830084Z 2022-03-17T20:53:29.8830664Z ---------------------------------------------------------------------- 2022-03-17T20:53:29.8831073Z Ran 1 test in 2.915s 2022-03-17T20:53:29.8831249Z 2022-03-17T20:53:29.8831432Z OK (skipped=1) 2022-03-17T20:53:29.8831674Z 2022-03-17T20:53:29.8831860Z Generating XML reports... 2022-03-17T20:53:29.8874388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205326.xml 2022-03-17T20:53:31.1670926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:31.1685249Z 2022-03-17T20:53:31.1685565Z Running tests... 2022-03-17T20:53:31.1686009Z ---------------------------------------------------------------------- 2022-03-17T20:53:32.7067897Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:32.7495625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24606 2022-03-17T20:53:32.7622107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24607 2022-03-17T20:53:33.8518259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:33.8518786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:33.8519591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:33.8520301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:33.8628993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:33.9529429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:34.1676176Z ok (2.999s) 2022-03-17T20:53:34.1676518Z 2022-03-17T20:53:34.1677054Z ---------------------------------------------------------------------- 2022-03-17T20:53:34.1677412Z Ran 1 test in 2.999s 2022-03-17T20:53:34.1677579Z 2022-03-17T20:53:34.1677685Z OK 2022-03-17T20:53:34.1677811Z 2022-03-17T20:53:34.1677942Z Generating XML reports... 2022-03-17T20:53:34.1722170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205331.xml 2022-03-17T20:53:35.4637179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:35.4651172Z 2022-03-17T20:53:35.4651492Z Running tests... 2022-03-17T20:53:35.4651951Z ---------------------------------------------------------------------- 2022-03-17T20:53:36.9926108Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:37.0356666Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24725 2022-03-17T20:53:37.0483956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24726 2022-03-17T20:53:38.1292779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:38.1293539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:38.1294352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:38.1295034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:38.1401161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:38.2304919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:38.4532177Z ok (2.988s) 2022-03-17T20:53:38.4532475Z 2022-03-17T20:53:38.4533053Z ---------------------------------------------------------------------- 2022-03-17T20:53:38.4533380Z Ran 1 test in 2.988s 2022-03-17T20:53:38.4533485Z 2022-03-17T20:53:38.4533574Z OK 2022-03-17T20:53:38.4533711Z 2022-03-17T20:53:38.4533829Z Generating XML reports... 2022-03-17T20:53:38.4584247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205335.xml 2022-03-17T20:53:39.7407978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:39.7421974Z 2022-03-17T20:53:39.7422127Z Running tests... 2022-03-17T20:53:39.7422836Z ---------------------------------------------------------------------- 2022-03-17T20:53:41.2696995Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:41.3122406Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24844 2022-03-17T20:53:41.3249439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24845 2022-03-17T20:53:42.3847995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:42.3848581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:42.3849295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:42.3849988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:42.3958040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:42.4859655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:42.7299037Z ok (2.987s) 2022-03-17T20:53:42.7299261Z 2022-03-17T20:53:42.7299650Z ---------------------------------------------------------------------- 2022-03-17T20:53:42.7300014Z Ran 1 test in 2.988s 2022-03-17T20:53:42.7300234Z 2022-03-17T20:53:42.7300519Z OK 2022-03-17T20:53:42.7300687Z 2022-03-17T20:53:42.7300839Z Generating XML reports... 2022-03-17T20:53:42.7344756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205339.xml 2022-03-17T20:53:44.0190560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:44.0204991Z 2022-03-17T20:53:44.0205212Z Running tests... 2022-03-17T20:53:44.0205985Z ---------------------------------------------------------------------- 2022-03-17T20:53:44.0226679Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:53:44.0227143Z 2022-03-17T20:53:44.0227570Z ---------------------------------------------------------------------- 2022-03-17T20:53:44.0227899Z Ran 1 test in 0.002s 2022-03-17T20:53:44.0228041Z 2022-03-17T20:53:44.0228148Z OK (skipped=1) 2022-03-17T20:53:44.0228303Z 2022-03-17T20:53:44.0228429Z Generating XML reports... 2022-03-17T20:53:44.0272746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205344.xml 2022-03-17T20:53:45.1222056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:45.1236116Z 2022-03-17T20:53:45.1236266Z Running tests... 2022-03-17T20:53:45.1236962Z ---------------------------------------------------------------------- 2022-03-17T20:53:45.1258130Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:53:45.1258443Z 2022-03-17T20:53:45.1258786Z ---------------------------------------------------------------------- 2022-03-17T20:53:45.1259064Z Ran 1 test in 0.002s 2022-03-17T20:53:45.1259215Z 2022-03-17T20:53:45.1259328Z OK (skipped=1) 2022-03-17T20:53:45.1259477Z 2022-03-17T20:53:45.1259606Z Generating XML reports... 2022-03-17T20:53:45.1303976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205345.xml 2022-03-17T20:53:46.2760709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:46.2774824Z 2022-03-17T20:53:46.2775079Z Running tests... 2022-03-17T20:53:46.2775986Z ---------------------------------------------------------------------- 2022-03-17T20:53:47.8304604Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:47.8730805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25033 2022-03-17T20:53:47.8856982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25034 2022-03-17T20:53:48.9801254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:48.9801799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:48.9802545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:48.9803238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:48.9809267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:48.9809739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:48.9917333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:53:48.9917892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:53:48.9918523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:53:48.9919203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:53:49.1902362Z ok (2.912s) 2022-03-17T20:53:49.1902565Z 2022-03-17T20:53:49.1903201Z ---------------------------------------------------------------------- 2022-03-17T20:53:49.1903623Z Ran 1 test in 2.913s 2022-03-17T20:53:49.1903808Z 2022-03-17T20:53:49.1903857Z OK 2022-03-17T20:53:49.1903994Z 2022-03-17T20:53:49.1904127Z Generating XML reports... 2022-03-17T20:53:49.1946152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205346.xml 2022-03-17T20:53:50.4736699Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:50.4751214Z 2022-03-17T20:53:50.4751633Z Running tests... 2022-03-17T20:53:50.4752107Z ---------------------------------------------------------------------- 2022-03-17T20:53:52.0310233Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:52.0738283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25158 2022-03-17T20:53:52.0868047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25159 2022-03-17T20:53:53.2139682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:53.2140640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:53.2141440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:53.2142137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:53.2147808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:53.2148879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:53.3914321Z skip (2.916s) 2022-03-17T20:53:53.3914798Z 2022-03-17T20:53:53.3915511Z ---------------------------------------------------------------------- 2022-03-17T20:53:53.3915874Z Ran 1 test in 2.916s 2022-03-17T20:53:53.3916042Z 2022-03-17T20:53:53.3916133Z OK (skipped=1) 2022-03-17T20:53:53.3916303Z 2022-03-17T20:53:53.3916428Z Generating XML reports... 2022-03-17T20:53:53.3958232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205350.xml 2022-03-17T20:53:54.6646333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:54.6659946Z 2022-03-17T20:53:54.6660202Z Running tests... 2022-03-17T20:53:54.6660922Z ---------------------------------------------------------------------- 2022-03-17T20:53:54.6682828Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:53:54.6683182Z 2022-03-17T20:53:54.6683785Z ---------------------------------------------------------------------- 2022-03-17T20:53:54.6684486Z Ran 1 test in 0.002s 2022-03-17T20:53:54.6684725Z 2022-03-17T20:53:54.6684816Z OK (skipped=1) 2022-03-17T20:53:54.6684971Z 2022-03-17T20:53:54.6685097Z Generating XML reports... 2022-03-17T20:53:54.6728085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205354.xml 2022-03-17T20:53:55.8053921Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:55.8068264Z 2022-03-17T20:53:55.8068722Z Running tests... 2022-03-17T20:53:55.8069229Z ---------------------------------------------------------------------- 2022-03-17T20:53:55.8090795Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:53:55.8091284Z 2022-03-17T20:53:55.8091726Z ---------------------------------------------------------------------- 2022-03-17T20:53:55.8092068Z Ran 1 test in 0.002s 2022-03-17T20:53:55.8092234Z 2022-03-17T20:53:55.8092344Z OK (skipped=1) 2022-03-17T20:53:55.8092484Z 2022-03-17T20:53:55.8092618Z Generating XML reports... 2022-03-17T20:53:55.8135702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205355.xml 2022-03-17T20:53:56.9567674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:53:56.9581260Z 2022-03-17T20:53:56.9581704Z Running tests... 2022-03-17T20:53:56.9582550Z ---------------------------------------------------------------------- 2022-03-17T20:53:58.4953732Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:53:58.5380972Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25343 2022-03-17T20:53:58.5508219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25344 2022-03-17T20:53:59.6668392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:53:59.6668877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:53:59.6669847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:59.6670583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:53:59.6676982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:53:59.6677767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:53:59.8556401Z ok (2.897s) 2022-03-17T20:53:59.8556796Z 2022-03-17T20:53:59.8557523Z ---------------------------------------------------------------------- 2022-03-17T20:53:59.8557965Z Ran 1 test in 2.897s 2022-03-17T20:53:59.8558148Z 2022-03-17T20:53:59.8558234Z OK 2022-03-17T20:53:59.8558368Z 2022-03-17T20:53:59.8558485Z Generating XML reports... 2022-03-17T20:53:59.8599520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205356.xml 2022-03-17T20:54:01.1453516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:01.1467547Z 2022-03-17T20:54:01.1467752Z Running tests... 2022-03-17T20:54:01.1468450Z ---------------------------------------------------------------------- 2022-03-17T20:54:02.7036365Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:02.7465925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25458 2022-03-17T20:54:02.7595847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25459 2022-03-17T20:54:03.8582546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:03.8583322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:03.8584174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:03.8584888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:03.8591061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:03.8592065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:03.8907486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:54:03.9009616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:54:03.9010879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:03.9011620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:03.9153368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:54:03.9154833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:54:03.9156173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:54:03.9256313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:54:03.9377200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T20:54:03.9378347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T20:54:03.9379031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:54:03.9379890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T20:54:04.1644782Z ok (3.017s) 2022-03-17T20:54:04.1645047Z 2022-03-17T20:54:04.1645642Z ---------------------------------------------------------------------- 2022-03-17T20:54:04.1646353Z Ran 1 test in 3.018s 2022-03-17T20:54:04.1646699Z 2022-03-17T20:54:04.1646881Z OK 2022-03-17T20:54:04.1647005Z 2022-03-17T20:54:04.1647140Z Generating XML reports... 2022-03-17T20:54:04.1688152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205401.xml 2022-03-17T20:54:05.4437554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:05.4452046Z 2022-03-17T20:54:05.4452601Z Running tests... 2022-03-17T20:54:05.4453100Z ---------------------------------------------------------------------- 2022-03-17T20:54:06.9881014Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:07.0308825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25597 2022-03-17T20:54:07.0438224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25598 2022-03-17T20:54:08.1101223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:08.1102070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:08.1102847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:08.1103537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:08.1209604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:08.2114139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:08.2225767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:54:08.2226561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:54:08.2227248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:08.2228013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:08.4486931Z ok (3.003s) 2022-03-17T20:54:08.4487195Z 2022-03-17T20:54:08.4487588Z ---------------------------------------------------------------------- 2022-03-17T20:54:08.4487928Z Ran 1 test in 3.003s 2022-03-17T20:54:08.4488097Z 2022-03-17T20:54:08.4488171Z OK 2022-03-17T20:54:08.4489426Z 2022-03-17T20:54:08.4490160Z Generating XML reports... 2022-03-17T20:54:08.4531366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205405.xml 2022-03-17T20:54:09.7374833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:09.7389544Z 2022-03-17T20:54:09.7389912Z Running tests... 2022-03-17T20:54:09.7390414Z ---------------------------------------------------------------------- 2022-03-17T20:54:11.2785590Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:11.3211684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25722 2022-03-17T20:54:11.3338010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25723 2022-03-17T20:54:12.4237804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:12.4238332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:12.4239343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:12.4240064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:12.4247426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:12.4247927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:12.4455837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:54:12.4456724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:54:12.4458086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:12.4459403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:12.6385456Z ok (2.899s) 2022-03-17T20:54:12.6385726Z 2022-03-17T20:54:12.6386167Z ---------------------------------------------------------------------- 2022-03-17T20:54:12.6386485Z Ran 1 test in 2.900s 2022-03-17T20:54:12.6386598Z 2022-03-17T20:54:12.6386651Z OK 2022-03-17T20:54:12.6386775Z 2022-03-17T20:54:12.6386913Z Generating XML reports... 2022-03-17T20:54:12.6428432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205409.xml 2022-03-17T20:54:13.9207254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:13.9221208Z 2022-03-17T20:54:13.9221632Z Running tests... 2022-03-17T20:54:13.9222103Z ---------------------------------------------------------------------- 2022-03-17T20:54:15.4677548Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:15.5111643Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25847 2022-03-17T20:54:15.5239611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25848 2022-03-17T20:54:16.6353215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:16.6353781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:16.6354495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:16.6355191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:16.6461618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:16.7365405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:16.7573892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:54:16.7574376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:54:16.7575085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:16.7575939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:16.9287108Z ok (3.006s) 2022-03-17T20:54:16.9287271Z 2022-03-17T20:54:16.9287834Z ---------------------------------------------------------------------- 2022-03-17T20:54:16.9288186Z Ran 1 test in 3.007s 2022-03-17T20:54:16.9288332Z 2022-03-17T20:54:16.9288405Z OK 2022-03-17T20:54:16.9288538Z 2022-03-17T20:54:16.9288669Z Generating XML reports... 2022-03-17T20:54:16.9331554Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205413.xml 2022-03-17T20:54:18.2095366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:18.2109704Z 2022-03-17T20:54:18.2109913Z Running tests... 2022-03-17T20:54:18.2110597Z ---------------------------------------------------------------------- 2022-03-17T20:54:19.7409563Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:19.7837591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25972 2022-03-17T20:54:19.7965487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25973 2022-03-17T20:54:20.8884454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:20.8885013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:20.8885775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:20.8886469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:20.8993281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:20.9895634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:21.0103304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:54:21.0103773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:54:21.0104501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:21.0105157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:54:21.2014180Z ok (2.990s) 2022-03-17T20:54:21.2014608Z 2022-03-17T20:54:21.2015269Z ---------------------------------------------------------------------- 2022-03-17T20:54:21.2015808Z Ran 1 test in 2.990s 2022-03-17T20:54:21.2016109Z 2022-03-17T20:54:21.2016264Z OK 2022-03-17T20:54:21.2016544Z 2022-03-17T20:54:21.2016781Z Generating XML reports... 2022-03-17T20:54:21.2061462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205418.xml 2022-03-17T20:54:22.4856377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:22.4870638Z 2022-03-17T20:54:22.4870873Z Running tests... 2022-03-17T20:54:22.4871341Z ---------------------------------------------------------------------- 2022-03-17T20:54:24.0194636Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:24.0623196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26097 2022-03-17T20:54:24.0751708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26098 2022-03-17T20:54:25.2003970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:25.2004727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:25.2005506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:25.2006206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:25.2012934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:25.2013371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:25.3798430Z skip (2.892s) 2022-03-17T20:54:25.3798586Z 2022-03-17T20:54:25.3799321Z ---------------------------------------------------------------------- 2022-03-17T20:54:25.3799698Z Ran 1 test in 2.893s 2022-03-17T20:54:25.3799865Z 2022-03-17T20:54:25.3799977Z OK (skipped=1) 2022-03-17T20:54:25.3800063Z 2022-03-17T20:54:25.3800156Z Generating XML reports... 2022-03-17T20:54:25.3842491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205422.xml 2022-03-17T20:54:26.6614615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:26.6628764Z 2022-03-17T20:54:26.6629143Z Running tests... 2022-03-17T20:54:26.6629618Z ---------------------------------------------------------------------- 2022-03-17T20:54:28.2171045Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:28.2596222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26212 2022-03-17T20:54:28.2725939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26213 2022-03-17T20:54:29.3879880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:29.3880427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:29.3881124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:29.3881815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:29.3990888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:29.4893770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:29.6773895Z skip (3.014s) 2022-03-17T20:54:29.6774111Z 2022-03-17T20:54:29.6774530Z ---------------------------------------------------------------------- 2022-03-17T20:54:29.6774848Z Ran 1 test in 3.014s 2022-03-17T20:54:29.6775016Z 2022-03-17T20:54:29.6775117Z OK (skipped=1) 2022-03-17T20:54:29.6775200Z 2022-03-17T20:54:29.6775310Z Generating XML reports... 2022-03-17T20:54:29.6818588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205426.xml 2022-03-17T20:54:30.9799544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:30.9814155Z 2022-03-17T20:54:30.9814471Z Running tests... 2022-03-17T20:54:30.9814904Z ---------------------------------------------------------------------- 2022-03-17T20:54:32.5539443Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:32.5969571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26327 2022-03-17T20:54:32.6098865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26328 2022-03-17T20:54:33.6728615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:33.6729159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:33.6730176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:33.6730879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:33.6739732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:33.6740579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:33.8148679Z skip (2.833s) 2022-03-17T20:54:33.8148927Z 2022-03-17T20:54:33.8149570Z ---------------------------------------------------------------------- 2022-03-17T20:54:33.8149862Z Ran 1 test in 2.833s 2022-03-17T20:54:33.8150028Z 2022-03-17T20:54:33.8150137Z OK (skipped=1) 2022-03-17T20:54:33.8150291Z 2022-03-17T20:54:33.8150436Z Generating XML reports... 2022-03-17T20:54:33.8193276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205430.xml 2022-03-17T20:54:35.1068341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:35.1083315Z 2022-03-17T20:54:35.1083617Z Running tests... 2022-03-17T20:54:35.1084037Z ---------------------------------------------------------------------- 2022-03-17T20:54:36.6525413Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:36.6955493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26442 2022-03-17T20:54:36.7084537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26443 2022-03-17T20:54:37.8113407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:37.8113927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:37.8114743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:37.8115436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:37.8224112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:37.9127944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:38.1133002Z skip (3.005s) 2022-03-17T20:54:38.1133189Z 2022-03-17T20:54:38.1133612Z ---------------------------------------------------------------------- 2022-03-17T20:54:38.1133971Z Ran 1 test in 3.005s 2022-03-17T20:54:38.1134135Z 2022-03-17T20:54:38.1134203Z OK (skipped=1) 2022-03-17T20:54:38.1134361Z 2022-03-17T20:54:38.1134492Z Generating XML reports... 2022-03-17T20:54:38.1179319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205435.xml 2022-03-17T20:54:39.3995598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:39.4009999Z 2022-03-17T20:54:39.4010417Z Running tests... 2022-03-17T20:54:39.4010875Z ---------------------------------------------------------------------- 2022-03-17T20:54:40.9307601Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:40.9737247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26557 2022-03-17T20:54:40.9865195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26558 2022-03-17T20:54:42.0928818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:42.0929361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:42.0930134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:42.0931070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:42.0937675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:42.0938238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:42.2912661Z ok (2.890s) 2022-03-17T20:54:42.2912983Z 2022-03-17T20:54:42.2913397Z ---------------------------------------------------------------------- 2022-03-17T20:54:42.2913714Z Ran 1 test in 2.890s 2022-03-17T20:54:42.2913878Z 2022-03-17T20:54:42.2914216Z OK 2022-03-17T20:54:42.2914368Z 2022-03-17T20:54:42.2914483Z Generating XML reports... 2022-03-17T20:54:42.2957201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205439.xml 2022-03-17T20:54:43.5673759Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:43.5687723Z 2022-03-17T20:54:43.5687876Z Running tests... 2022-03-17T20:54:43.5688570Z ---------------------------------------------------------------------- 2022-03-17T20:54:45.0980954Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:45.1409712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26676 2022-03-17T20:54:45.1540275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26677 2022-03-17T20:54:46.2825034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:46.2825559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:46.2826347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:46.2827047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:46.2934451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:46.3836184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:46.5587513Z ok (2.990s) 2022-03-17T20:54:46.5587695Z 2022-03-17T20:54:46.5588129Z ---------------------------------------------------------------------- 2022-03-17T20:54:46.5588446Z Ran 1 test in 2.990s 2022-03-17T20:54:46.5588609Z 2022-03-17T20:54:46.5588704Z OK 2022-03-17T20:54:46.5588838Z 2022-03-17T20:54:46.5588944Z Generating XML reports... 2022-03-17T20:54:46.5630779Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205443.xml 2022-03-17T20:54:47.7972501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:47.7986921Z 2022-03-17T20:54:47.7987242Z Running tests... 2022-03-17T20:54:47.7987668Z ---------------------------------------------------------------------- 2022-03-17T20:54:49.3386690Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:49.3816658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26791 2022-03-17T20:54:49.3945284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26792 2022-03-17T20:54:50.4571665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:50.4572171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:50.4572957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:50.4573633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:50.4680422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:50.5582796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:50.7993446Z ok (3.000s) 2022-03-17T20:54:50.7993923Z 2022-03-17T20:54:50.7994610Z ---------------------------------------------------------------------- 2022-03-17T20:54:50.7994965Z Ran 1 test in 3.001s 2022-03-17T20:54:50.7995137Z 2022-03-17T20:54:50.7995241Z OK 2022-03-17T20:54:50.7995377Z 2022-03-17T20:54:50.7995508Z Generating XML reports... 2022-03-17T20:54:50.8037553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205447.xml 2022-03-17T20:54:52.0885761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:52.0900027Z 2022-03-17T20:54:52.0900173Z Running tests... 2022-03-17T20:54:52.0900908Z ---------------------------------------------------------------------- 2022-03-17T20:54:53.6352081Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:53.6779308Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26910 2022-03-17T20:54:53.6908642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26911 2022-03-17T20:54:54.8095731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:54.8096293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:54.8097098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:54.8097790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:54.8206095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:54.9108143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:55.0957155Z ok (3.005s) 2022-03-17T20:54:55.0957362Z 2022-03-17T20:54:55.0957735Z ---------------------------------------------------------------------- 2022-03-17T20:54:55.0958071Z Ran 1 test in 3.006s 2022-03-17T20:54:55.0958239Z 2022-03-17T20:54:55.0958332Z OK 2022-03-17T20:54:55.0958456Z 2022-03-17T20:54:55.0958592Z Generating XML reports... 2022-03-17T20:54:55.1000876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205452.xml 2022-03-17T20:54:56.3801879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:54:56.3816052Z 2022-03-17T20:54:56.3816525Z Running tests... 2022-03-17T20:54:56.3817058Z ---------------------------------------------------------------------- 2022-03-17T20:54:57.9393980Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:54:57.9825293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27029 2022-03-17T20:54:57.9954207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27030 2022-03-17T20:54:59.0598406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:54:59.0598947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:54:59.0599726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:59.0600417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:54:59.0706954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:54:59.1609374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:54:59.4004783Z ok (3.018s) 2022-03-17T20:54:59.4004976Z 2022-03-17T20:54:59.4005433Z ---------------------------------------------------------------------- 2022-03-17T20:54:59.4005760Z Ran 1 test in 3.019s 2022-03-17T20:54:59.4005971Z 2022-03-17T20:54:59.4006012Z OK 2022-03-17T20:54:59.4007484Z 2022-03-17T20:54:59.4008038Z Generating XML reports... 2022-03-17T20:54:59.4052275Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205456.xml 2022-03-17T20:55:00.7012115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:00.7026764Z 2022-03-17T20:55:00.7027200Z Running tests... 2022-03-17T20:55:00.7027670Z ---------------------------------------------------------------------- 2022-03-17T20:55:02.2407400Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:02.2837196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27148 2022-03-17T20:55:02.2965018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27149 2022-03-17T20:55:03.3658016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:03.3658568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:03.3659272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:03.3659967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:03.3767946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:03.4669047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:03.6010626Z ok (2.898s) 2022-03-17T20:55:03.6010864Z 2022-03-17T20:55:03.6011252Z ---------------------------------------------------------------------- 2022-03-17T20:55:03.6011534Z Ran 1 test in 2.898s 2022-03-17T20:55:03.6011680Z 2022-03-17T20:55:03.6011770Z OK 2022-03-17T20:55:03.6011903Z 2022-03-17T20:55:03.6012035Z Generating XML reports... 2022-03-17T20:55:03.6054598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205500.xml 2022-03-17T20:55:04.8889277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:04.8903807Z 2022-03-17T20:55:04.8904263Z Running tests... 2022-03-17T20:55:04.8904970Z ---------------------------------------------------------------------- 2022-03-17T20:55:06.4288899Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:06.4719771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27263 2022-03-17T20:55:06.4848505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27264 2022-03-17T20:55:07.6056295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:07.6056807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:07.6057597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:07.6058286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:07.6165184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:07.7068061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:07.7179426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:55:07.7179961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:55:07.7180944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:55:07.7181618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:55:07.8895500Z ok (2.999s) 2022-03-17T20:55:07.8896194Z 2022-03-17T20:55:07.8896873Z ---------------------------------------------------------------------- 2022-03-17T20:55:07.8897192Z Ran 1 test in 2.999s 2022-03-17T20:55:07.8897358Z 2022-03-17T20:55:07.8897451Z OK 2022-03-17T20:55:07.8897586Z 2022-03-17T20:55:07.8897700Z Generating XML reports... 2022-03-17T20:55:07.8940023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205504.xml 2022-03-17T20:55:09.1841836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:09.1855859Z 2022-03-17T20:55:09.1856166Z Running tests... 2022-03-17T20:55:09.1856607Z ---------------------------------------------------------------------- 2022-03-17T20:55:10.7294417Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:10.7723763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27388 2022-03-17T20:55:10.7852731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27389 2022-03-17T20:55:11.8853680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:11.8854236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:11.8855019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:11.8855720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:11.8963332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:11.9865521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:12.0073828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:55:12.0074352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:55:12.0075025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:55:12.0075721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:55:12.1901273Z ok (3.004s) 2022-03-17T20:55:12.1901497Z 2022-03-17T20:55:12.1901889Z ---------------------------------------------------------------------- 2022-03-17T20:55:12.1902207Z Ran 1 test in 3.004s 2022-03-17T20:55:12.1902372Z 2022-03-17T20:55:12.1902473Z OK 2022-03-17T20:55:12.1902607Z 2022-03-17T20:55:12.1902740Z Generating XML reports... 2022-03-17T20:55:12.1945807Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205509.xml 2022-03-17T20:55:13.4722557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:13.4736797Z 2022-03-17T20:55:13.4737185Z Running tests... 2022-03-17T20:55:13.4737680Z ---------------------------------------------------------------------- 2022-03-17T20:55:14.9854147Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:15.0278575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27513 2022-03-17T20:55:15.0405694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27514 2022-03-17T20:55:16.1714138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:16.1714994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:16.1715813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:16.1716915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:16.1722316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:16.1723007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:16.1830613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:55:16.1831410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:55:16.1832038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:55:16.1833068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:55:16.4451468Z ok (2.971s) 2022-03-17T20:55:16.4451695Z 2022-03-17T20:55:16.4452102Z ---------------------------------------------------------------------- 2022-03-17T20:55:16.4452441Z Ran 1 test in 2.971s 2022-03-17T20:55:16.4452622Z 2022-03-17T20:55:16.4452709Z OK 2022-03-17T20:55:16.4452853Z 2022-03-17T20:55:16.4452983Z Generating XML reports... 2022-03-17T20:55:16.4504357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205513.xml 2022-03-17T20:55:17.7311146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:17.7325478Z 2022-03-17T20:55:17.7325835Z Running tests... 2022-03-17T20:55:17.7326253Z ---------------------------------------------------------------------- 2022-03-17T20:55:19.2738975Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:19.3165339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27638 2022-03-17T20:55:19.3292591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27639 2022-03-17T20:55:20.4205637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:20.4206107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:20.4206925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:20.4207615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:20.4313347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:20.5216662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:20.5326699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:55:20.5327419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:55:20.5328221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:55:20.5328894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:55:20.7341673Z ok (3.001s) 2022-03-17T20:55:20.7341973Z 2022-03-17T20:55:20.7342336Z ---------------------------------------------------------------------- 2022-03-17T20:55:20.7342674Z Ran 1 test in 3.002s 2022-03-17T20:55:20.7342850Z 2022-03-17T20:55:20.7342929Z OK 2022-03-17T20:55:20.7343055Z 2022-03-17T20:55:20.7343195Z Generating XML reports... 2022-03-17T20:55:20.7385860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205517.xml 2022-03-17T20:55:22.0281056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:22.0295681Z 2022-03-17T20:55:22.0296114Z Running tests... 2022-03-17T20:55:22.0296612Z ---------------------------------------------------------------------- 2022-03-17T20:55:23.5666569Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:23.6093673Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27763 2022-03-17T20:55:23.6221753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27764 2022-03-17T20:55:24.7295115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:24.7295677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:24.7296397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:24.7297111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:24.7303007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:24.7303980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:24.9268560Z skip (2.897s) 2022-03-17T20:55:24.9268745Z 2022-03-17T20:55:24.9269135Z ---------------------------------------------------------------------- 2022-03-17T20:55:24.9269510Z Ran 1 test in 2.897s 2022-03-17T20:55:24.9269677Z 2022-03-17T20:55:24.9269785Z OK (skipped=1) 2022-03-17T20:55:24.9269935Z 2022-03-17T20:55:24.9270044Z Generating XML reports... 2022-03-17T20:55:24.9313288Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205522.xml 2022-03-17T20:55:26.2035268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:26.2049754Z 2022-03-17T20:55:26.2050033Z Running tests... 2022-03-17T20:55:26.2050453Z ---------------------------------------------------------------------- 2022-03-17T20:55:27.7547921Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:27.7977801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27878 2022-03-17T20:55:27.8105780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27879 2022-03-17T20:55:28.8751389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:28.8751922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:28.8752707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:28.8753423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:28.8859536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:28.9765649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:29.1153371Z skip (2.910s) 2022-03-17T20:55:29.1153841Z 2022-03-17T20:55:29.1154269Z ---------------------------------------------------------------------- 2022-03-17T20:55:29.1154610Z Ran 1 test in 2.910s 2022-03-17T20:55:29.1154777Z 2022-03-17T20:55:29.1154869Z OK (skipped=1) 2022-03-17T20:55:29.1155025Z 2022-03-17T20:55:29.1155153Z Generating XML reports... 2022-03-17T20:55:29.1198232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205526.xml 2022-03-17T20:55:30.4006802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:30.4021765Z 2022-03-17T20:55:30.4021971Z Running tests... 2022-03-17T20:55:30.4022709Z ---------------------------------------------------------------------- 2022-03-17T20:55:31.9534419Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:31.9963340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27993 2022-03-17T20:55:32.0092986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27994 2022-03-17T20:55:33.0676602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:33.0677619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:33.0678395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:33.0679092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:33.0787518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:33.1692014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:33.3141274Z skip (2.911s) 2022-03-17T20:55:33.3141707Z 2022-03-17T20:55:33.3142322Z ---------------------------------------------------------------------- 2022-03-17T20:55:33.3142665Z Ran 1 test in 2.912s 2022-03-17T20:55:33.3142831Z 2022-03-17T20:55:33.3142941Z OK (skipped=1) 2022-03-17T20:55:33.3143098Z 2022-03-17T20:55:33.3143226Z Generating XML reports... 2022-03-17T20:55:33.3187093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205530.xml 2022-03-17T20:55:34.6013350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:34.6028272Z 2022-03-17T20:55:34.6028730Z Running tests... 2022-03-17T20:55:34.6029263Z ---------------------------------------------------------------------- 2022-03-17T20:55:36.1394277Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:36.1823409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28108 2022-03-17T20:55:36.1952264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28109 2022-03-17T20:55:37.2585660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:37.2586241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:37.2587055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:37.2587803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:37.2595481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:37.2595981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:37.3997922Z skip (2.797s) 2022-03-17T20:55:37.3998238Z 2022-03-17T20:55:37.3998624Z ---------------------------------------------------------------------- 2022-03-17T20:55:37.3999306Z Ran 1 test in 2.797s 2022-03-17T20:55:37.3999465Z 2022-03-17T20:55:37.3999569Z OK (skipped=1) 2022-03-17T20:55:37.3999725Z 2022-03-17T20:55:37.3999812Z Generating XML reports... 2022-03-17T20:55:37.4043184Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205534.xml 2022-03-17T20:55:38.6790361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:38.6805324Z 2022-03-17T20:55:38.6805600Z Running tests... 2022-03-17T20:55:38.6806294Z ---------------------------------------------------------------------- 2022-03-17T20:55:40.2222203Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:40.2651501Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28223 2022-03-17T20:55:40.2778021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28224 2022-03-17T20:55:41.3517446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:41.3517995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:41.3518768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:41.3519474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:41.3626071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:41.4528329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:41.6826887Z ok (3.002s) 2022-03-17T20:55:41.6827096Z 2022-03-17T20:55:41.6827494Z ---------------------------------------------------------------------- 2022-03-17T20:55:41.6827830Z Ran 1 test in 3.002s 2022-03-17T20:55:41.6827996Z 2022-03-17T20:55:41.6828084Z OK 2022-03-17T20:55:41.6828200Z 2022-03-17T20:55:41.6828330Z Generating XML reports... 2022-03-17T20:55:41.6871634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205538.xml 2022-03-17T20:55:42.9715630Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:42.9730228Z 2022-03-17T20:55:42.9730359Z Running tests... 2022-03-17T20:55:42.9731085Z ---------------------------------------------------------------------- 2022-03-17T20:55:44.5021235Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:44.5448081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28342 2022-03-17T20:55:44.5576510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28343 2022-03-17T20:55:45.6460715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:45.6461556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:45.6462323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:45.6463014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:45.6569100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:45.7473242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:45.9621849Z ok (2.989s) 2022-03-17T20:55:45.9622071Z 2022-03-17T20:55:45.9622490Z ---------------------------------------------------------------------- 2022-03-17T20:55:45.9622812Z Ran 1 test in 2.989s 2022-03-17T20:55:45.9622979Z 2022-03-17T20:55:45.9623073Z OK 2022-03-17T20:55:45.9623480Z 2022-03-17T20:55:45.9623601Z Generating XML reports... 2022-03-17T20:55:45.9667547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205542.xml 2022-03-17T20:55:47.2451424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:47.2465711Z 2022-03-17T20:55:47.2465862Z Running tests... 2022-03-17T20:55:47.2466572Z ---------------------------------------------------------------------- 2022-03-17T20:55:48.7772646Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:48.8198146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28461 2022-03-17T20:55:48.8327236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28462 2022-03-17T20:55:49.9451298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:49.9451804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:49.9452606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:49.9453292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:49.9459222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:49.9459717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:51.9406909Z ok (4.694s) 2022-03-17T20:55:51.9407213Z 2022-03-17T20:55:51.9407631Z ---------------------------------------------------------------------- 2022-03-17T20:55:51.9407919Z Ran 1 test in 4.694s 2022-03-17T20:55:51.9408053Z 2022-03-17T20:55:51.9408144Z OK 2022-03-17T20:55:51.9408259Z 2022-03-17T20:55:51.9408388Z Generating XML reports... 2022-03-17T20:55:51.9452097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205547.xml 2022-03-17T20:55:53.2389151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:53.2403687Z 2022-03-17T20:55:53.2403828Z Running tests... 2022-03-17T20:55:53.2404676Z ---------------------------------------------------------------------- 2022-03-17T20:55:54.7804494Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:55:54.8226996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28582 2022-03-17T20:55:54.8353313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28583 2022-03-17T20:55:55.9334859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:55:55.9335887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:55:55.9336693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:55.9337381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:55:55.9444204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:55:56.0351003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:55:57.9434754Z ok (4.703s) 2022-03-17T20:55:57.9435170Z 2022-03-17T20:55:57.9435979Z ---------------------------------------------------------------------- 2022-03-17T20:55:57.9436426Z Ran 1 test in 4.703s 2022-03-17T20:55:57.9436575Z 2022-03-17T20:55:57.9436667Z OK 2022-03-17T20:55:57.9436801Z 2022-03-17T20:55:57.9436936Z Generating XML reports... 2022-03-17T20:55:57.9479017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205553.xml 2022-03-17T20:55:59.2471053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:55:59.2485577Z 2022-03-17T20:55:59.2485854Z Running tests... 2022-03-17T20:55:59.2486770Z ---------------------------------------------------------------------- 2022-03-17T20:56:00.8060920Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:56:00.8487416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28703 2022-03-17T20:56:00.8616011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28704 2022-03-17T20:56:01.9503935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:01.9504466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:01.9505281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:01.9506035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:01.9512203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:01.9512989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:02.1662940Z ok (2.917s) 2022-03-17T20:56:02.1663223Z 2022-03-17T20:56:02.1663902Z ---------------------------------------------------------------------- 2022-03-17T20:56:02.1664280Z Ran 1 test in 2.918s 2022-03-17T20:56:02.1664471Z 2022-03-17T20:56:02.1664564Z OK 2022-03-17T20:56:02.1664681Z 2022-03-17T20:56:02.1664813Z Generating XML reports... 2022-03-17T20:56:02.1706754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205559.xml 2022-03-17T20:56:03.4621361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:03.4636379Z 2022-03-17T20:56:03.4636593Z Running tests... 2022-03-17T20:56:03.4637369Z ---------------------------------------------------------------------- 2022-03-17T20:56:04.9985581Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:56:05.0411287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28822 2022-03-17T20:56:05.0537277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28823 2022-03-17T20:56:06.1648163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:06.1648687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:06.1649391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:06.1650092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:06.1757786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:06.2661941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:07.8612531Z ok (4.397s) 2022-03-17T20:56:07.8612884Z 2022-03-17T20:56:07.8613529Z ---------------------------------------------------------------------- 2022-03-17T20:56:07.8614077Z Ran 1 test in 4.398s 2022-03-17T20:56:07.8614363Z 2022-03-17T20:56:07.8614509Z OK 2022-03-17T20:56:07.8614731Z 2022-03-17T20:56:07.8614998Z Generating XML reports... 2022-03-17T20:56:07.8657960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205603.xml 2022-03-17T20:56:09.1656535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:09.1670507Z 2022-03-17T20:56:09.1670792Z Running tests... 2022-03-17T20:56:09.1671714Z ---------------------------------------------------------------------- 2022-03-17T20:56:10.7293227Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:56:10.7721626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28939 2022-03-17T20:56:10.7851264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28940 2022-03-17T20:56:11.9041682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:11.9042398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:11.9043243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:11.9043998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:11.9149963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:12.0052963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:12.1898562Z ok (3.022s) 2022-03-17T20:56:12.1898848Z 2022-03-17T20:56:12.1899227Z ---------------------------------------------------------------------- 2022-03-17T20:56:12.1899546Z Ran 1 test in 3.023s 2022-03-17T20:56:12.1899707Z 2022-03-17T20:56:12.1899773Z OK 2022-03-17T20:56:12.1899918Z 2022-03-17T20:56:12.1900086Z Generating XML reports... 2022-03-17T20:56:12.1943824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205609.xml 2022-03-17T20:56:13.4763553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:13.4777703Z 2022-03-17T20:56:13.4778114Z Running tests... 2022-03-17T20:56:13.4778586Z ---------------------------------------------------------------------- 2022-03-17T20:56:15.0130909Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:56:15.0559688Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29058 2022-03-17T20:56:15.0687816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29059 2022-03-17T20:56:16.1876996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:16.1878032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:16.1878826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:16.1879495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:16.1988191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:16.2888616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:16.4736481Z ok (2.996s) 2022-03-17T20:56:16.4736684Z 2022-03-17T20:56:16.4737093Z ---------------------------------------------------------------------- 2022-03-17T20:56:16.4737432Z Ran 1 test in 2.996s 2022-03-17T20:56:16.4737568Z 2022-03-17T20:56:16.4737648Z OK 2022-03-17T20:56:16.4737764Z 2022-03-17T20:56:16.4737898Z Generating XML reports... 2022-03-17T20:56:16.4782086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205613.xml 2022-03-17T20:56:17.7615978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:17.7630946Z 2022-03-17T20:56:17.7631208Z Running tests... 2022-03-17T20:56:17.7631953Z ---------------------------------------------------------------------- 2022-03-17T20:56:19.3004083Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:56:19.3435508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29177 2022-03-17T20:56:19.3563365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29178 2022-03-17T20:56:20.4091225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:20.4091803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:20.4092799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:20.4093494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:20.4199260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:20.5101596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:20.6609293Z ok (2.898s) 2022-03-17T20:56:20.6609539Z 2022-03-17T20:56:20.6609935Z ---------------------------------------------------------------------- 2022-03-17T20:56:20.6610273Z Ran 1 test in 2.898s 2022-03-17T20:56:20.6610384Z 2022-03-17T20:56:20.6610479Z OK 2022-03-17T20:56:20.6610612Z 2022-03-17T20:56:20.6610748Z Generating XML reports... 2022-03-17T20:56:20.6654362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205617.xml 2022-03-17T20:56:21.9506247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:21.9519929Z 2022-03-17T20:56:21.9520330Z Running tests... 2022-03-17T20:56:21.9520837Z ---------------------------------------------------------------------- 2022-03-17T20:56:23.4891850Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:56:23.5315466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29296 2022-03-17T20:56:23.5442802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29297 2022-03-17T20:56:24.6374624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:24.6375188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:24.6375921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:24.6376612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:24.6383768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:24.6384263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:28.6554936Z ok (6.703s) 2022-03-17T20:56:28.6555331Z 2022-03-17T20:56:28.6555931Z ---------------------------------------------------------------------- 2022-03-17T20:56:28.6556205Z Ran 1 test in 6.703s 2022-03-17T20:56:28.6556371Z 2022-03-17T20:56:28.6556469Z OK 2022-03-17T20:56:28.6556586Z 2022-03-17T20:56:28.6556728Z Generating XML reports... 2022-03-17T20:56:28.6598928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205621.xml 2022-03-17T20:56:29.9568930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:29.9583433Z 2022-03-17T20:56:29.9583887Z Running tests... 2022-03-17T20:56:29.9584360Z ---------------------------------------------------------------------- 2022-03-17T20:56:31.4968351Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:56:31.5395599Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29419 2022-03-17T20:56:31.5524178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29420 2022-03-17T20:56:32.6698077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:32.6698613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:32.6699381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:32.6700606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:32.6808609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:32.7711717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:36.7640701Z ok (6.805s) 2022-03-17T20:56:36.7640979Z 2022-03-17T20:56:36.7641373Z ---------------------------------------------------------------------- 2022-03-17T20:56:36.7641695Z Ran 1 test in 6.806s 2022-03-17T20:56:36.7641811Z 2022-03-17T20:56:36.7641922Z OK 2022-03-17T20:56:36.7642035Z 2022-03-17T20:56:36.7642152Z Generating XML reports... 2022-03-17T20:56:36.7685406Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205629.xml 2022-03-17T20:56:38.0642732Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:38.0656679Z 2022-03-17T20:56:38.0656893Z Running tests... 2022-03-17T20:56:38.0657302Z ---------------------------------------------------------------------- 2022-03-17T20:56:39.6084987Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:56:39.6510955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29542 2022-03-17T20:56:39.6638776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29543 2022-03-17T20:56:40.7755160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:56:40.7755669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:56:40.7756406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:40.7757104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:56:40.7866099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:56:40.8770879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:56:44.8758807Z ok (6.810s) 2022-03-17T20:56:44.8759217Z 2022-03-17T20:56:44.8759840Z ---------------------------------------------------------------------- 2022-03-17T20:56:44.8760461Z Ran 1 test in 6.810s 2022-03-17T20:56:44.8760741Z 2022-03-17T20:56:44.8760897Z OK 2022-03-17T20:56:44.8761145Z 2022-03-17T20:56:44.8761394Z Generating XML reports... 2022-03-17T20:56:44.8804448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205638.xml 2022-03-17T20:56:46.1857830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:46.1872888Z 2022-03-17T20:56:46.1873279Z Running tests... 2022-03-17T20:56:46.1874016Z ---------------------------------------------------------------------- 2022-03-17T20:56:46.1893104Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:46.1893562Z 2022-03-17T20:56:46.1893958Z ---------------------------------------------------------------------- 2022-03-17T20:56:46.1894578Z Ran 1 test in 0.002s 2022-03-17T20:56:46.1894801Z 2022-03-17T20:56:46.1894998Z OK (skipped=1) 2022-03-17T20:56:46.1895244Z 2022-03-17T20:56:46.1895373Z Generating XML reports... 2022-03-17T20:56:46.1939826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205646.xml 2022-03-17T20:56:47.3381670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:47.3395824Z 2022-03-17T20:56:47.3396199Z Running tests... 2022-03-17T20:56:47.3396685Z ---------------------------------------------------------------------- 2022-03-17T20:56:47.3416090Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:47.3416444Z 2022-03-17T20:56:47.3416693Z ---------------------------------------------------------------------- 2022-03-17T20:56:47.3417004Z Ran 1 test in 0.002s 2022-03-17T20:56:47.3417160Z 2022-03-17T20:56:47.3417255Z OK (skipped=1) 2022-03-17T20:56:47.3417387Z 2022-03-17T20:56:47.3417596Z Generating XML reports... 2022-03-17T20:56:47.3461902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205647.xml 2022-03-17T20:56:48.4925434Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:48.4939617Z 2022-03-17T20:56:48.4939797Z Running tests... 2022-03-17T20:56:48.4940223Z ---------------------------------------------------------------------- 2022-03-17T20:56:48.4960620Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:48.4960891Z 2022-03-17T20:56:48.4961248Z ---------------------------------------------------------------------- 2022-03-17T20:56:48.4961510Z Ran 1 test in 0.002s 2022-03-17T20:56:48.4961676Z 2022-03-17T20:56:48.4961792Z OK (skipped=1) 2022-03-17T20:56:48.4961939Z 2022-03-17T20:56:48.4962050Z Generating XML reports... 2022-03-17T20:56:48.5006530Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205648.xml 2022-03-17T20:56:49.6467866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:49.6481851Z 2022-03-17T20:56:49.6482301Z Running tests... 2022-03-17T20:56:49.6483034Z ---------------------------------------------------------------------- 2022-03-17T20:56:49.6503756Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:49.6503982Z 2022-03-17T20:56:49.6504304Z ---------------------------------------------------------------------- 2022-03-17T20:56:49.6504636Z Ran 1 test in 0.002s 2022-03-17T20:56:49.6504805Z 2022-03-17T20:56:49.6504923Z OK (skipped=1) 2022-03-17T20:56:49.6505081Z 2022-03-17T20:56:49.6505206Z Generating XML reports... 2022-03-17T20:56:49.6549924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205649.xml 2022-03-17T20:56:50.7939361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:50.7954045Z 2022-03-17T20:56:50.7954213Z Running tests... 2022-03-17T20:56:50.7954658Z ---------------------------------------------------------------------- 2022-03-17T20:56:50.7973972Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:50.7974281Z 2022-03-17T20:56:50.7974572Z ---------------------------------------------------------------------- 2022-03-17T20:56:50.7974867Z Ran 1 test in 0.002s 2022-03-17T20:56:50.7975013Z 2022-03-17T20:56:50.7975123Z OK (skipped=1) 2022-03-17T20:56:50.7975273Z 2022-03-17T20:56:50.7975412Z Generating XML reports... 2022-03-17T20:56:50.8020095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205650.xml 2022-03-17T20:56:51.9500099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:51.9513807Z 2022-03-17T20:56:51.9514172Z Running tests... 2022-03-17T20:56:51.9514682Z ---------------------------------------------------------------------- 2022-03-17T20:56:51.9535136Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:51.9535401Z 2022-03-17T20:56:51.9535768Z ---------------------------------------------------------------------- 2022-03-17T20:56:51.9536026Z Ran 1 test in 0.002s 2022-03-17T20:56:51.9536206Z 2022-03-17T20:56:51.9536295Z OK (skipped=1) 2022-03-17T20:56:51.9536450Z 2022-03-17T20:56:51.9536569Z Generating XML reports... 2022-03-17T20:56:51.9581378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205651.xml 2022-03-17T20:56:53.1018195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:53.1032376Z 2022-03-17T20:56:53.1032583Z Running tests... 2022-03-17T20:56:53.1033046Z ---------------------------------------------------------------------- 2022-03-17T20:56:53.1052759Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:53.1053058Z 2022-03-17T20:56:53.1053334Z ---------------------------------------------------------------------- 2022-03-17T20:56:53.1053666Z Ran 1 test in 0.002s 2022-03-17T20:56:53.1053845Z 2022-03-17T20:56:53.1054040Z OK (skipped=1) 2022-03-17T20:56:53.1054295Z 2022-03-17T20:56:53.1054461Z Generating XML reports... 2022-03-17T20:56:53.1098214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205653.xml 2022-03-17T20:56:54.2575195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:54.2590347Z 2022-03-17T20:56:54.2590455Z Running tests... 2022-03-17T20:56:54.2591325Z ---------------------------------------------------------------------- 2022-03-17T20:56:54.2611518Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:54.2611770Z 2022-03-17T20:56:54.2612068Z ---------------------------------------------------------------------- 2022-03-17T20:56:54.2612642Z Ran 1 test in 0.002s 2022-03-17T20:56:54.2612791Z 2022-03-17T20:56:54.2612899Z OK (skipped=1) 2022-03-17T20:56:54.2613054Z 2022-03-17T20:56:54.2614112Z Generating XML reports... 2022-03-17T20:56:54.2657815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205654.xml 2022-03-17T20:56:55.4093659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:55.4107902Z 2022-03-17T20:56:55.4108329Z Running tests... 2022-03-17T20:56:55.4109014Z ---------------------------------------------------------------------- 2022-03-17T20:56:55.4127882Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:55.4128157Z 2022-03-17T20:56:55.4128575Z ---------------------------------------------------------------------- 2022-03-17T20:56:55.4129107Z Ran 1 test in 0.002s 2022-03-17T20:56:55.4129278Z 2022-03-17T20:56:55.4129375Z OK (skipped=1) 2022-03-17T20:56:55.4129492Z 2022-03-17T20:56:55.4129624Z Generating XML reports... 2022-03-17T20:56:55.4173590Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205655.xml 2022-03-17T20:56:56.5674157Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:56.5689002Z 2022-03-17T20:56:56.5689308Z Running tests... 2022-03-17T20:56:56.5689744Z ---------------------------------------------------------------------- 2022-03-17T20:56:56.5709547Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:56.5709856Z 2022-03-17T20:56:56.5710170Z ---------------------------------------------------------------------- 2022-03-17T20:56:56.5710749Z Ran 1 test in 0.002s 2022-03-17T20:56:56.5710911Z 2022-03-17T20:56:56.5711001Z OK (skipped=1) 2022-03-17T20:56:56.5711919Z 2022-03-17T20:56:56.5712553Z Generating XML reports... 2022-03-17T20:56:56.5755415Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205656.xml 2022-03-17T20:56:57.7196702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:57.7210840Z 2022-03-17T20:56:57.7211274Z Running tests... 2022-03-17T20:56:57.7211939Z ---------------------------------------------------------------------- 2022-03-17T20:56:57.7232574Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:57.7233157Z 2022-03-17T20:56:57.7233468Z ---------------------------------------------------------------------- 2022-03-17T20:56:57.7233730Z Ran 1 test in 0.002s 2022-03-17T20:56:57.7233850Z 2022-03-17T20:56:57.7233979Z OK (skipped=1) 2022-03-17T20:56:57.7234274Z 2022-03-17T20:56:57.7234489Z Generating XML reports... 2022-03-17T20:56:57.7277932Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205657.xml 2022-03-17T20:56:58.8700633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:56:58.8715425Z 2022-03-17T20:56:58.8715669Z Running tests... 2022-03-17T20:56:58.8716282Z ---------------------------------------------------------------------- 2022-03-17T20:56:58.8736677Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:56:58.8737029Z 2022-03-17T20:56:58.8737492Z ---------------------------------------------------------------------- 2022-03-17T20:56:58.8737869Z Ran 1 test in 0.002s 2022-03-17T20:56:58.8738041Z 2022-03-17T20:56:58.8738159Z OK (skipped=1) 2022-03-17T20:56:58.8738308Z 2022-03-17T20:56:58.8738386Z Generating XML reports... 2022-03-17T20:56:58.8782540Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205658.xml 2022-03-17T20:57:00.0211415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:00.0225932Z 2022-03-17T20:57:00.0226194Z Running tests... 2022-03-17T20:57:00.0226756Z ---------------------------------------------------------------------- 2022-03-17T20:57:00.0246074Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:00.0246452Z 2022-03-17T20:57:00.0246940Z ---------------------------------------------------------------------- 2022-03-17T20:57:00.0247277Z Ran 1 test in 0.002s 2022-03-17T20:57:00.0247436Z 2022-03-17T20:57:00.0247502Z OK (skipped=1) 2022-03-17T20:57:00.0247657Z 2022-03-17T20:57:00.0247766Z Generating XML reports... 2022-03-17T20:57:00.0292500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205700.xml 2022-03-17T20:57:01.1405806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:01.1421891Z 2022-03-17T20:57:01.1422443Z Running tests... 2022-03-17T20:57:01.1422934Z ---------------------------------------------------------------------- 2022-03-17T20:57:01.1444515Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:01.1444819Z 2022-03-17T20:57:01.1445555Z ---------------------------------------------------------------------- 2022-03-17T20:57:01.1446250Z Ran 1 test in 0.002s 2022-03-17T20:57:01.1446563Z 2022-03-17T20:57:01.1446763Z OK (skipped=1) 2022-03-17T20:57:01.1447063Z 2022-03-17T20:57:01.1447314Z Generating XML reports... 2022-03-17T20:57:01.1494072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205701.xml 2022-03-17T20:57:02.3002882Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:02.3017109Z 2022-03-17T20:57:02.3017225Z Running tests... 2022-03-17T20:57:02.3017623Z ---------------------------------------------------------------------- 2022-03-17T20:57:02.3037679Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:02.3037917Z 2022-03-17T20:57:02.3038210Z ---------------------------------------------------------------------- 2022-03-17T20:57:02.3038533Z Ran 1 test in 0.002s 2022-03-17T20:57:02.3038695Z 2022-03-17T20:57:02.3038770Z OK (skipped=1) 2022-03-17T20:57:02.3038907Z 2022-03-17T20:57:02.3039031Z Generating XML reports... 2022-03-17T20:57:02.3083666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205702.xml 2022-03-17T20:57:03.4551523Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:03.4565755Z 2022-03-17T20:57:03.4565946Z Running tests... 2022-03-17T20:57:03.4566608Z ---------------------------------------------------------------------- 2022-03-17T20:57:03.4587964Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:03.4588245Z 2022-03-17T20:57:03.4588539Z ---------------------------------------------------------------------- 2022-03-17T20:57:03.4588880Z Ran 1 test in 0.002s 2022-03-17T20:57:03.4589012Z 2022-03-17T20:57:03.4589129Z OK (skipped=1) 2022-03-17T20:57:03.4589295Z 2022-03-17T20:57:03.4589333Z Generating XML reports... 2022-03-17T20:57:03.4633621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205703.xml 2022-03-17T20:57:04.6089521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:04.6103875Z 2022-03-17T20:57:04.6104155Z Running tests... 2022-03-17T20:57:04.6104841Z ---------------------------------------------------------------------- 2022-03-17T20:57:04.6124077Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:04.6124353Z 2022-03-17T20:57:04.6124712Z ---------------------------------------------------------------------- 2022-03-17T20:57:04.6125213Z Ran 1 test in 0.002s 2022-03-17T20:57:04.6125359Z 2022-03-17T20:57:04.6125467Z OK (skipped=1) 2022-03-17T20:57:04.6125619Z 2022-03-17T20:57:04.6125724Z Generating XML reports... 2022-03-17T20:57:04.6169460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205704.xml 2022-03-17T20:57:05.7491997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:05.7507253Z 2022-03-17T20:57:05.7507596Z Running tests... 2022-03-17T20:57:05.7508031Z ---------------------------------------------------------------------- 2022-03-17T20:57:05.7527849Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:05.7528418Z 2022-03-17T20:57:05.7528720Z ---------------------------------------------------------------------- 2022-03-17T20:57:05.7529033Z Ran 1 test in 0.002s 2022-03-17T20:57:05.7529195Z 2022-03-17T20:57:05.7529301Z OK (skipped=1) 2022-03-17T20:57:05.7529457Z 2022-03-17T20:57:05.7529568Z Generating XML reports... 2022-03-17T20:57:05.7573806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205705.xml 2022-03-17T20:57:06.8508518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:06.8523410Z 2022-03-17T20:57:06.8523754Z Running tests... 2022-03-17T20:57:06.8524444Z ---------------------------------------------------------------------- 2022-03-17T20:57:06.8546324Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:06.8546649Z 2022-03-17T20:57:06.8547315Z ---------------------------------------------------------------------- 2022-03-17T20:57:06.8547784Z Ran 1 test in 0.002s 2022-03-17T20:57:06.8547951Z 2022-03-17T20:57:06.8548067Z OK (skipped=1) 2022-03-17T20:57:06.8548222Z 2022-03-17T20:57:06.8548328Z Generating XML reports... 2022-03-17T20:57:06.8594115Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205706.xml 2022-03-17T20:57:07.9699921Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:07.9715678Z 2022-03-17T20:57:07.9715983Z Running tests... 2022-03-17T20:57:07.9716698Z ---------------------------------------------------------------------- 2022-03-17T20:57:07.9738111Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:07.9738444Z 2022-03-17T20:57:07.9738818Z ---------------------------------------------------------------------- 2022-03-17T20:57:07.9739120Z Ran 1 test in 0.002s 2022-03-17T20:57:07.9739286Z 2022-03-17T20:57:07.9739397Z OK (skipped=1) 2022-03-17T20:57:07.9739552Z 2022-03-17T20:57:07.9739659Z Generating XML reports... 2022-03-17T20:57:07.9784767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205707.xml 2022-03-17T20:57:09.1188852Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:09.1203477Z 2022-03-17T20:57:09.1203764Z Running tests... 2022-03-17T20:57:09.1204182Z ---------------------------------------------------------------------- 2022-03-17T20:57:09.1224383Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:09.1224648Z 2022-03-17T20:57:09.1224960Z ---------------------------------------------------------------------- 2022-03-17T20:57:09.1225271Z Ran 1 test in 0.002s 2022-03-17T20:57:09.1225432Z 2022-03-17T20:57:09.1225543Z OK (skipped=1) 2022-03-17T20:57:09.1225658Z 2022-03-17T20:57:09.1225767Z Generating XML reports... 2022-03-17T20:57:09.1269967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205709.xml 2022-03-17T20:57:10.2792641Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:10.2806805Z 2022-03-17T20:57:10.2807075Z Running tests... 2022-03-17T20:57:10.2807612Z ---------------------------------------------------------------------- 2022-03-17T20:57:10.2828872Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:10.2829122Z 2022-03-17T20:57:10.2829433Z ---------------------------------------------------------------------- 2022-03-17T20:57:10.2829761Z Ran 1 test in 0.002s 2022-03-17T20:57:10.2829928Z 2022-03-17T20:57:10.2830034Z OK (skipped=1) 2022-03-17T20:57:10.2830150Z 2022-03-17T20:57:10.2830272Z Generating XML reports... 2022-03-17T20:57:10.2874550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205710.xml 2022-03-17T20:57:11.4366544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:11.4381110Z 2022-03-17T20:57:11.4381601Z Running tests... 2022-03-17T20:57:11.4382220Z ---------------------------------------------------------------------- 2022-03-17T20:57:11.4402141Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:11.4402401Z 2022-03-17T20:57:11.4402781Z ---------------------------------------------------------------------- 2022-03-17T20:57:11.4403318Z Ran 1 test in 0.002s 2022-03-17T20:57:11.4403486Z 2022-03-17T20:57:11.4403601Z OK (skipped=1) 2022-03-17T20:57:11.4403732Z 2022-03-17T20:57:11.4403840Z Generating XML reports... 2022-03-17T20:57:11.4448221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205711.xml 2022-03-17T20:57:12.5871861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:12.5885739Z 2022-03-17T20:57:12.5886056Z Running tests... 2022-03-17T20:57:12.5886495Z ---------------------------------------------------------------------- 2022-03-17T20:57:12.5907626Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:12.5907935Z 2022-03-17T20:57:12.5908192Z ---------------------------------------------------------------------- 2022-03-17T20:57:12.5908520Z Ran 1 test in 0.002s 2022-03-17T20:57:12.5908666Z 2022-03-17T20:57:12.5908747Z OK (skipped=1) 2022-03-17T20:57:12.5909177Z 2022-03-17T20:57:12.5909328Z Generating XML reports... 2022-03-17T20:57:12.5952959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205712.xml 2022-03-17T20:57:13.7409362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:13.7423925Z 2022-03-17T20:57:13.7424016Z Running tests... 2022-03-17T20:57:13.7425107Z ---------------------------------------------------------------------- 2022-03-17T20:57:15.3090941Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:15.3524146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30505 2022-03-17T20:57:15.3659247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30506 2022-03-17T20:57:16.4811562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:16.4812134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:16.4812874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:16.4813578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:16.4920033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:16.5826332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:18.7979902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:57:18.7980750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:57:18.7981567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:57:18.7982260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:57:19.1749209Z ok (5.432s) 2022-03-17T20:57:19.1749504Z 2022-03-17T20:57:19.1749861Z ---------------------------------------------------------------------- 2022-03-17T20:57:19.1750163Z Ran 1 test in 5.432s 2022-03-17T20:57:19.1750274Z 2022-03-17T20:57:19.1750365Z OK 2022-03-17T20:57:19.1750500Z 2022-03-17T20:57:19.1750614Z Generating XML reports... 2022-03-17T20:57:19.1791765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205713.xml 2022-03-17T20:57:20.4848271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:20.4862280Z 2022-03-17T20:57:20.4862440Z Running tests... 2022-03-17T20:57:20.4863086Z ---------------------------------------------------------------------- 2022-03-17T20:57:22.0206772Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:22.0633713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30634 2022-03-17T20:57:22.0760483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30635 2022-03-17T20:57:23.1934929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:23.1935427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:23.1936177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:23.1936840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:23.1943499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:23.1944025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:23.3806643Z skip (2.894s) 2022-03-17T20:57:23.3806818Z 2022-03-17T20:57:23.3807194Z ---------------------------------------------------------------------- 2022-03-17T20:57:23.3807556Z Ran 1 test in 2.894s 2022-03-17T20:57:23.3807720Z 2022-03-17T20:57:23.3807827Z OK (skipped=1) 2022-03-17T20:57:23.3807981Z 2022-03-17T20:57:23.3808088Z Generating XML reports... 2022-03-17T20:57:23.3850912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205720.xml 2022-03-17T20:57:24.6650324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:24.6665865Z 2022-03-17T20:57:24.6666147Z Running tests... 2022-03-17T20:57:24.6666538Z ---------------------------------------------------------------------- 2022-03-17T20:57:24.6687102Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:24.6687970Z 2022-03-17T20:57:24.6688258Z ---------------------------------------------------------------------- 2022-03-17T20:57:24.6688615Z Ran 1 test in 0.002s 2022-03-17T20:57:24.6688776Z 2022-03-17T20:57:24.6688884Z OK (skipped=1) 2022-03-17T20:57:24.6689052Z 2022-03-17T20:57:24.6689160Z Generating XML reports... 2022-03-17T20:57:24.6734669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205724.xml 2022-03-17T20:57:25.7778402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:25.7795014Z 2022-03-17T20:57:25.7795464Z Running tests... 2022-03-17T20:57:25.7795949Z ---------------------------------------------------------------------- 2022-03-17T20:57:27.3205888Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:27.3636414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30784 2022-03-17T20:57:27.3764614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30785 2022-03-17T20:57:28.4859375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:28.4860780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:28.4862241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:28.4863576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:28.4869987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:28.4870478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:29.4825542Z ok (3.703s) 2022-03-17T20:57:29.4825773Z 2022-03-17T20:57:29.4826175Z ---------------------------------------------------------------------- 2022-03-17T20:57:29.4826484Z Ran 1 test in 3.703s 2022-03-17T20:57:29.4826649Z 2022-03-17T20:57:29.4826742Z OK 2022-03-17T20:57:29.4826860Z 2022-03-17T20:57:29.4826995Z Generating XML reports... 2022-03-17T20:57:29.4870263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205725.xml 2022-03-17T20:57:30.7861135Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:30.7875456Z 2022-03-17T20:57:30.7875825Z Running tests... 2022-03-17T20:57:30.7876335Z ---------------------------------------------------------------------- 2022-03-17T20:57:32.3251448Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:32.3681972Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30899 2022-03-17T20:57:32.3810528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30900 2022-03-17T20:57:33.4807997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:33.4808577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:33.4809278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:33.4809953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:33.4816100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:33.4816998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:35.8896356Z ok (5.102s) 2022-03-17T20:57:35.8896612Z 2022-03-17T20:57:35.8897028Z ---------------------------------------------------------------------- 2022-03-17T20:57:35.8897372Z Ran 1 test in 5.102s 2022-03-17T20:57:35.8897493Z 2022-03-17T20:57:35.8897533Z OK 2022-03-17T20:57:35.8897667Z 2022-03-17T20:57:35.8897800Z Generating XML reports... 2022-03-17T20:57:35.8940086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205730.xml 2022-03-17T20:57:37.1891512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:37.1905953Z 2022-03-17T20:57:37.1906201Z Running tests... 2022-03-17T20:57:37.1906644Z ---------------------------------------------------------------------- 2022-03-17T20:57:38.7431681Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:38.7860091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31016 2022-03-17T20:57:38.7986618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31017 2022-03-17T20:57:39.8948028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:39.8948570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:39.8949350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:39.8950049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:39.9056236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:39.9958686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:40.0073006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:57:40.0073506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:57:40.0074199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:57:40.0074892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:57:41.0046472Z ok (3.814s) 2022-03-17T20:57:41.0046693Z 2022-03-17T20:57:41.0047066Z ---------------------------------------------------------------------- 2022-03-17T20:57:41.0047408Z Ran 1 test in 3.814s 2022-03-17T20:57:41.0047573Z 2022-03-17T20:57:41.0049174Z OK 2022-03-17T20:57:41.0049371Z 2022-03-17T20:57:41.0049698Z Generating XML reports... 2022-03-17T20:57:41.0091522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205737.xml 2022-03-17T20:57:42.3029242Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:42.3043991Z 2022-03-17T20:57:42.3044434Z Running tests... 2022-03-17T20:57:42.3044901Z ---------------------------------------------------------------------- 2022-03-17T20:57:43.8510968Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:43.8938765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31137 2022-03-17T20:57:43.9065824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31138 2022-03-17T20:57:44.9581952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:44.9582494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:44.9583283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:44.9583980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:44.9592376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:44.9592821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:45.1110804Z skip (2.806s) 2022-03-17T20:57:45.1111030Z 2022-03-17T20:57:45.1111410Z ---------------------------------------------------------------------- 2022-03-17T20:57:45.1111730Z Ran 1 test in 2.807s 2022-03-17T20:57:45.1111875Z 2022-03-17T20:57:45.1112006Z OK (skipped=1) 2022-03-17T20:57:45.1112162Z 2022-03-17T20:57:45.1112311Z Generating XML reports... 2022-03-17T20:57:45.1155325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205742.xml 2022-03-17T20:57:46.3469323Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:46.3484161Z 2022-03-17T20:57:46.3484594Z Running tests... 2022-03-17T20:57:46.3485120Z ---------------------------------------------------------------------- 2022-03-17T20:57:47.8940139Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:47.9366554Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31252 2022-03-17T20:57:47.9493836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31253 2022-03-17T20:57:49.0524269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:49.0524849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:49.0525874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:49.0526579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:49.0532881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:49.0533582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:49.2539614Z skip (2.905s) 2022-03-17T20:57:49.2540009Z 2022-03-17T20:57:49.2541061Z ---------------------------------------------------------------------- 2022-03-17T20:57:49.2541433Z Ran 1 test in 2.906s 2022-03-17T20:57:49.2541582Z 2022-03-17T20:57:49.2541709Z OK (skipped=1) 2022-03-17T20:57:49.2541865Z 2022-03-17T20:57:49.2541992Z Generating XML reports... 2022-03-17T20:57:49.2585495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205746.xml 2022-03-17T20:57:50.5432080Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:50.5446767Z 2022-03-17T20:57:50.5447398Z Running tests... 2022-03-17T20:57:50.5448176Z ---------------------------------------------------------------------- 2022-03-17T20:57:52.0794221Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:52.1222517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31367 2022-03-17T20:57:52.1351525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31368 2022-03-17T20:57:53.1937887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:53.1938418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:53.1939200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:53.1939892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:53.1947072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:53.1948616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:53.3396205Z skip (2.795s) 2022-03-17T20:57:53.3396411Z 2022-03-17T20:57:53.3396764Z ---------------------------------------------------------------------- 2022-03-17T20:57:53.3397112Z Ran 1 test in 2.795s 2022-03-17T20:57:53.3397279Z 2022-03-17T20:57:53.3397371Z OK (skipped=1) 2022-03-17T20:57:53.3397508Z 2022-03-17T20:57:53.3397631Z Generating XML reports... 2022-03-17T20:57:53.3440780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205750.xml 2022-03-17T20:57:54.6134523Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:54.6148885Z 2022-03-17T20:57:54.6149050Z Running tests... 2022-03-17T20:57:54.6149729Z ---------------------------------------------------------------------- 2022-03-17T20:57:56.1204498Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:57:56.1624819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31482 2022-03-17T20:57:56.1750177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31483 2022-03-17T20:57:57.2479467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:57:57.2480012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:57:57.2480756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:57.2481447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:57:57.2587656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:57:57.3491557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:57:57.3701852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:57:57.3702347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:57:57.3703244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:57:57.3703934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:57:58.6817489Z ok (4.066s) 2022-03-17T20:57:58.6817683Z 2022-03-17T20:57:58.6818091Z ---------------------------------------------------------------------- 2022-03-17T20:57:58.6818437Z Ran 1 test in 4.067s 2022-03-17T20:57:58.6818602Z 2022-03-17T20:57:58.6818686Z OK 2022-03-17T20:57:58.6818803Z 2022-03-17T20:57:58.6818943Z Generating XML reports... 2022-03-17T20:57:58.6864309Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205754.xml 2022-03-17T20:57:59.9851612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:57:59.9866813Z 2022-03-17T20:57:59.9867180Z Running tests... 2022-03-17T20:57:59.9867645Z ---------------------------------------------------------------------- 2022-03-17T20:57:59.9891682Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:57:59.9892593Z 2022-03-17T20:57:59.9892931Z ---------------------------------------------------------------------- 2022-03-17T20:57:59.9893260Z Ran 1 test in 0.003s 2022-03-17T20:57:59.9893444Z 2022-03-17T20:57:59.9893550Z OK (skipped=1) 2022-03-17T20:57:59.9893693Z 2022-03-17T20:57:59.9893818Z Generating XML reports... 2022-03-17T20:57:59.9939450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205759.xml 2022-03-17T20:58:01.1403005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:01.1417593Z 2022-03-17T20:58:01.1417915Z Running tests... 2022-03-17T20:58:01.1418348Z ---------------------------------------------------------------------- 2022-03-17T20:58:02.6921354Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:02.7354802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31638 2022-03-17T20:58:02.7485739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31639 2022-03-17T20:58:03.8730474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:03.8733172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:03.8733974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:03.8834413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:03.8841617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:03.9748530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:04.1535334Z skip (3.011s) 2022-03-17T20:58:04.1535628Z 2022-03-17T20:58:04.1536025Z ---------------------------------------------------------------------- 2022-03-17T20:58:04.1536414Z Ran 1 test in 3.012s 2022-03-17T20:58:04.1536525Z 2022-03-17T20:58:04.1536618Z OK (skipped=1) 2022-03-17T20:58:04.1537671Z 2022-03-17T20:58:04.1537846Z Generating XML reports... 2022-03-17T20:58:04.1580039Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205801.xml 2022-03-17T20:58:05.4407645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:05.4421210Z 2022-03-17T20:58:05.4421340Z Running tests... 2022-03-17T20:58:05.4422104Z ---------------------------------------------------------------------- 2022-03-17T20:58:06.9636830Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:07.0057971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31753 2022-03-17T20:58:07.0185682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31754 2022-03-17T20:58:08.1191830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:08.1192323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:08.1193294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:08.1194022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:08.1199532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:08.1200490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:08.3231941Z ok (2.881s) 2022-03-17T20:58:08.3232228Z 2022-03-17T20:58:08.3232543Z ---------------------------------------------------------------------- 2022-03-17T20:58:08.3233082Z Ran 1 test in 2.881s 2022-03-17T20:58:08.3233374Z 2022-03-17T20:58:08.3233393Z OK 2022-03-17T20:58:08.3233503Z 2022-03-17T20:58:08.3233667Z Generating XML reports... 2022-03-17T20:58:08.3276577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205805.xml 2022-03-17T20:58:09.6083369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:09.6098406Z 2022-03-17T20:58:09.6098577Z Running tests... 2022-03-17T20:58:09.6099053Z ---------------------------------------------------------------------- 2022-03-17T20:58:11.1689828Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:11.2118328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31868 2022-03-17T20:58:11.2249833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31869 2022-03-17T20:58:12.3093578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:12.3094128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:12.3094898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:12.3095598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:12.3201909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:12.4105498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:12.6297195Z ok (3.020s) 2022-03-17T20:58:12.6297401Z 2022-03-17T20:58:12.6297806Z ---------------------------------------------------------------------- 2022-03-17T20:58:12.6298120Z Ran 1 test in 3.020s 2022-03-17T20:58:12.6298283Z 2022-03-17T20:58:12.6298378Z OK 2022-03-17T20:58:12.6298545Z 2022-03-17T20:58:12.6298661Z Generating XML reports... 2022-03-17T20:58:12.6341446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205809.xml 2022-03-17T20:58:13.9230006Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:13.9245041Z 2022-03-17T20:58:13.9245398Z Running tests... 2022-03-17T20:58:13.9245794Z ---------------------------------------------------------------------- 2022-03-17T20:58:13.9272959Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:58:13.9273732Z 2022-03-17T20:58:13.9274388Z ---------------------------------------------------------------------- 2022-03-17T20:58:13.9274746Z Ran 1 test in 0.003s 2022-03-17T20:58:13.9274912Z 2022-03-17T20:58:13.9275004Z OK (skipped=1) 2022-03-17T20:58:13.9275162Z 2022-03-17T20:58:13.9275238Z Generating XML reports... 2022-03-17T20:58:13.9320206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205813.xml 2022-03-17T20:58:15.0673105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:15.0687522Z 2022-03-17T20:58:15.0687657Z Running tests... 2022-03-17T20:58:15.0688702Z ---------------------------------------------------------------------- 2022-03-17T20:58:15.0718109Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:58:15.0718621Z 2022-03-17T20:58:15.0718953Z ---------------------------------------------------------------------- 2022-03-17T20:58:15.0719300Z Ran 1 test in 0.003s 2022-03-17T20:58:15.0719464Z 2022-03-17T20:58:15.0719538Z OK (skipped=1) 2022-03-17T20:58:15.0719677Z 2022-03-17T20:58:15.0719809Z Generating XML reports... 2022-03-17T20:58:15.0764889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205815.xml 2022-03-17T20:58:16.2311683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:16.2333535Z 2022-03-17T20:58:16.2333714Z Running tests... 2022-03-17T20:58:16.2334228Z ---------------------------------------------------------------------- 2022-03-17T20:58:16.2364083Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:58:16.2364408Z 2022-03-17T20:58:16.2364693Z ---------------------------------------------------------------------- 2022-03-17T20:58:16.2365015Z Ran 1 test in 0.003s 2022-03-17T20:58:16.2365144Z 2022-03-17T20:58:16.2365231Z OK (skipped=1) 2022-03-17T20:58:16.2365401Z 2022-03-17T20:58:16.2365530Z Generating XML reports... 2022-03-17T20:58:16.2411403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205816.xml 2022-03-17T20:58:17.3881831Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:17.3895955Z 2022-03-17T20:58:17.3896113Z Running tests... 2022-03-17T20:58:17.3896803Z ---------------------------------------------------------------------- 2022-03-17T20:58:17.3921120Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:58:17.3921376Z 2022-03-17T20:58:17.3921695Z ---------------------------------------------------------------------- 2022-03-17T20:58:17.3922230Z Ran 1 test in 0.003s 2022-03-17T20:58:17.3922410Z 2022-03-17T20:58:17.3922522Z OK (skipped=1) 2022-03-17T20:58:17.3922650Z 2022-03-17T20:58:17.3922775Z Generating XML reports... 2022-03-17T20:58:17.3967076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205817.xml 2022-03-17T20:58:18.5431260Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:18.5445698Z 2022-03-17T20:58:18.5446075Z Running tests... 2022-03-17T20:58:18.5446566Z ---------------------------------------------------------------------- 2022-03-17T20:58:18.5467951Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:58:18.5468426Z 2022-03-17T20:58:18.5468763Z ---------------------------------------------------------------------- 2022-03-17T20:58:18.5469089Z Ran 1 test in 0.002s 2022-03-17T20:58:18.5469246Z 2022-03-17T20:58:18.5469337Z OK (skipped=1) 2022-03-17T20:58:18.5469488Z 2022-03-17T20:58:18.5469611Z Generating XML reports... 2022-03-17T20:58:18.5514131Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205818.xml 2022-03-17T20:58:19.6954985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:19.6969028Z 2022-03-17T20:58:19.6969234Z Running tests... 2022-03-17T20:58:19.6969682Z ---------------------------------------------------------------------- 2022-03-17T20:58:19.6997093Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T20:58:19.6997387Z 2022-03-17T20:58:19.6997698Z ---------------------------------------------------------------------- 2022-03-17T20:58:19.6998198Z Ran 1 test in 0.003s 2022-03-17T20:58:19.6998362Z 2022-03-17T20:58:19.6998448Z OK (skipped=1) 2022-03-17T20:58:19.6998592Z 2022-03-17T20:58:19.6998947Z Generating XML reports... 2022-03-17T20:58:19.7043144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205819.xml 2022-03-17T20:58:20.8491569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:20.8506552Z 2022-03-17T20:58:20.8506811Z Running tests... 2022-03-17T20:58:20.8507236Z ---------------------------------------------------------------------- 2022-03-17T20:58:20.8529882Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T20:58:20.8530991Z 2022-03-17T20:58:20.8531449Z ---------------------------------------------------------------------- 2022-03-17T20:58:20.8531835Z Ran 1 test in 0.002s 2022-03-17T20:58:20.8531947Z 2022-03-17T20:58:20.8532055Z OK (skipped=1) 2022-03-17T20:58:20.8532213Z 2022-03-17T20:58:20.8532319Z Generating XML reports... 2022-03-17T20:58:20.8578008Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205820.xml 2022-03-17T20:58:21.9978157Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:21.9993219Z 2022-03-17T20:58:21.9993692Z Running tests... 2022-03-17T20:58:21.9994204Z ---------------------------------------------------------------------- 2022-03-17T20:58:23.5647206Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:23.6075319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32228 2022-03-17T20:58:23.6205027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32229 2022-03-17T20:58:24.7557660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:24.7558662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:24.7560147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:24.7561446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:24.7566949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:24.7567888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:25.0253501Z ok (3.026s) 2022-03-17T20:58:25.0253679Z 2022-03-17T20:58:25.0254065Z ---------------------------------------------------------------------- 2022-03-17T20:58:25.0254398Z Ran 1 test in 3.026s 2022-03-17T20:58:25.0254566Z 2022-03-17T20:58:25.0254996Z OK 2022-03-17T20:58:25.0255139Z 2022-03-17T20:58:25.0255551Z Generating XML reports... 2022-03-17T20:58:25.0299371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205821.xml 2022-03-17T20:58:26.3050366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:26.3064563Z 2022-03-17T20:58:26.3064981Z Running tests... 2022-03-17T20:58:26.3065482Z ---------------------------------------------------------------------- 2022-03-17T20:58:27.8228370Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:27.8654656Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32347 2022-03-17T20:58:27.8781500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32348 2022-03-17T20:58:28.9862713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:28.9863260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:28.9864218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:28.9864940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:28.9970508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:29.0876499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:31.0861997Z ok (4.779s) 2022-03-17T20:58:31.0862173Z 2022-03-17T20:58:31.0862604Z ---------------------------------------------------------------------- 2022-03-17T20:58:31.0862934Z Ran 1 test in 4.780s 2022-03-17T20:58:31.0863078Z 2022-03-17T20:58:31.0863162Z OK 2022-03-17T20:58:31.0863278Z 2022-03-17T20:58:31.0863409Z Generating XML reports... 2022-03-17T20:58:31.0906395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205826.xml 2022-03-17T20:58:32.3854634Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:32.3869040Z 2022-03-17T20:58:32.3869230Z Running tests... 2022-03-17T20:58:32.3869677Z ---------------------------------------------------------------------- 2022-03-17T20:58:33.9342592Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:33.9772135Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32468 2022-03-17T20:58:33.9902272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32469 2022-03-17T20:58:35.0918259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:35.0918857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:35.0919639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:35.0920333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:35.0926825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:35.0927303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:35.1034613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:58:35.1035115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:58:35.1035790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:35.1036453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:35.3949581Z ok (3.008s) 2022-03-17T20:58:35.3949805Z 2022-03-17T20:58:35.3950209Z ---------------------------------------------------------------------- 2022-03-17T20:58:35.3950526Z Ran 1 test in 3.008s 2022-03-17T20:58:35.3950689Z 2022-03-17T20:58:35.3950785Z OK 2022-03-17T20:58:35.3950919Z 2022-03-17T20:58:35.3951036Z Generating XML reports... 2022-03-17T20:58:35.3993063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205832.xml 2022-03-17T20:58:36.6830186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:36.6844063Z 2022-03-17T20:58:36.6844283Z Running tests... 2022-03-17T20:58:36.6844737Z ---------------------------------------------------------------------- 2022-03-17T20:58:38.2368098Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:38.2800924Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32593 2022-03-17T20:58:38.2930936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32594 2022-03-17T20:58:39.4230221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:39.4230757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:39.4231524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:39.4232222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:39.4238306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:39.4238915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:39.5977353Z skip (2.913s) 2022-03-17T20:58:39.5977554Z 2022-03-17T20:58:39.5977968Z ---------------------------------------------------------------------- 2022-03-17T20:58:39.5978366Z Ran 1 test in 2.913s 2022-03-17T20:58:39.5978506Z 2022-03-17T20:58:39.5978628Z OK (skipped=1) 2022-03-17T20:58:39.5978755Z 2022-03-17T20:58:39.5978845Z Generating XML reports... 2022-03-17T20:58:39.6022201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205836.xml 2022-03-17T20:58:40.8811775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:40.8826496Z 2022-03-17T20:58:40.8826733Z Running tests... 2022-03-17T20:58:40.8827220Z ---------------------------------------------------------------------- 2022-03-17T20:58:42.4220905Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:42.4648228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32708 2022-03-17T20:58:42.4775519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32709 2022-03-17T20:58:43.5576990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:43.5577555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:43.5578343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:43.5579026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:43.5687661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:43.6592447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:45.2848012Z ok (4.402s) 2022-03-17T20:58:45.2848228Z 2022-03-17T20:58:45.2848667Z ---------------------------------------------------------------------- 2022-03-17T20:58:45.2848987Z Ran 1 test in 4.402s 2022-03-17T20:58:45.2849158Z 2022-03-17T20:58:45.2849243Z OK 2022-03-17T20:58:45.2849372Z 2022-03-17T20:58:45.2849473Z Generating XML reports... 2022-03-17T20:58:45.2894321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205840.xml 2022-03-17T20:58:46.5874121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:46.5889340Z 2022-03-17T20:58:46.5889578Z Running tests... 2022-03-17T20:58:46.5890048Z ---------------------------------------------------------------------- 2022-03-17T20:58:48.1167465Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:48.1595832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32825 2022-03-17T20:58:48.1722941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32826 2022-03-17T20:58:49.2888634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:49.2889332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:49.2890123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:49.2890811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:49.2897693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:49.2898180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:49.4769674Z ok (2.888s) 2022-03-17T20:58:49.4769908Z 2022-03-17T20:58:49.4770321Z ---------------------------------------------------------------------- 2022-03-17T20:58:49.4770648Z Ran 1 test in 2.888s 2022-03-17T20:58:49.4770797Z 2022-03-17T20:58:49.4770892Z OK 2022-03-17T20:58:49.4771036Z 2022-03-17T20:58:49.4771189Z Generating XML reports... 2022-03-17T20:58:49.4814412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205846.xml 2022-03-17T20:58:50.7186284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:50.7200941Z 2022-03-17T20:58:50.7201188Z Running tests... 2022-03-17T20:58:50.7201628Z ---------------------------------------------------------------------- 2022-03-17T20:58:52.2570570Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:52.3000543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32940 2022-03-17T20:58:52.3131194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32941 2022-03-17T20:58:53.4203735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:53.4204581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:53.4205327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:53.4206026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:53.4312131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:53.5214717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:53.5421783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:58:53.5422458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:58:53.5423175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:53.5423865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:53.5630200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:58:53.5631140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:58:53.5631827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:58:53.5632520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:58:55.2204972Z ok (4.500s) 2022-03-17T20:58:55.2205160Z 2022-03-17T20:58:55.2205556Z ---------------------------------------------------------------------- 2022-03-17T20:58:55.2205897Z Ran 1 test in 4.500s 2022-03-17T20:58:55.2206074Z 2022-03-17T20:58:55.2206404Z OK 2022-03-17T20:58:55.2206560Z 2022-03-17T20:58:55.2206695Z Generating XML reports... 2022-03-17T20:58:55.2248697Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205850.xml 2022-03-17T20:58:56.5172612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:58:56.5187180Z 2022-03-17T20:58:56.5187700Z Running tests... 2022-03-17T20:58:56.5188191Z ---------------------------------------------------------------------- 2022-03-17T20:58:58.0548503Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:58:58.0977841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33069 2022-03-17T20:58:58.1106006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33070 2022-03-17T20:58:59.2172656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:58:59.2173172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:58:59.2173890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:59.2174544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:58:59.2180973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:58:59.2182105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:58:59.2289532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T20:58:59.2290289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T20:58:59.2290900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:59.2291567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T20:58:59.2398953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T20:58:59.2399502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T20:58:59.2400091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:58:59.2400771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T20:59:00.9180172Z ok (4.399s) 2022-03-17T20:59:00.9180703Z 2022-03-17T20:59:00.9181107Z ---------------------------------------------------------------------- 2022-03-17T20:59:00.9181471Z Ran 1 test in 4.399s 2022-03-17T20:59:00.9181644Z 2022-03-17T20:59:00.9181680Z OK 2022-03-17T20:59:00.9181784Z 2022-03-17T20:59:00.9181919Z Generating XML reports... 2022-03-17T20:59:00.9224183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205856.xml 2022-03-17T20:59:02.2256730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:02.2272867Z 2022-03-17T20:59:02.2273091Z Running tests... 2022-03-17T20:59:02.2273904Z ---------------------------------------------------------------------- 2022-03-17T20:59:03.7847936Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:03.8301956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33198 2022-03-17T20:59:03.8439765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33199 2022-03-17T20:59:04.9275685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:04.9276251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:04.9277030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:04.9277743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:04.9283770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:04.9284263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:06.5495685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:06.5496246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:06.8529024Z ok (4.625s) 2022-03-17T20:59:06.8529261Z 2022-03-17T20:59:06.8529652Z ---------------------------------------------------------------------- 2022-03-17T20:59:06.8529990Z Ran 1 test in 4.626s 2022-03-17T20:59:06.8530160Z 2022-03-17T20:59:06.8530255Z OK 2022-03-17T20:59:06.8530370Z 2022-03-17T20:59:06.8530522Z Generating XML reports... 2022-03-17T20:59:06.8573447Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205902.xml 2022-03-17T20:59:08.1156546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:08.1171955Z 2022-03-17T20:59:08.1172230Z Running tests... 2022-03-17T20:59:08.1172687Z ---------------------------------------------------------------------- 2022-03-17T20:59:09.6512747Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:09.6942708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33319 2022-03-17T20:59:09.7071282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33320 2022-03-17T20:59:10.8183015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:10.8183592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:10.8184396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:10.8185083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:10.8191925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:10.8192406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:12.4519282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:12.4519795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:12.4530802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:12.4531281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:12.7149008Z ok (4.597s) 2022-03-17T20:59:12.7149370Z 2022-03-17T20:59:12.7150000Z ---------------------------------------------------------------------- 2022-03-17T20:59:12.7150596Z Ran 1 test in 4.598s 2022-03-17T20:59:12.7150879Z 2022-03-17T20:59:12.7151037Z OK 2022-03-17T20:59:12.7151298Z 2022-03-17T20:59:12.7151509Z Generating XML reports... 2022-03-17T20:59:12.7194530Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205908.xml 2022-03-17T20:59:14.0110584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:14.0124498Z 2022-03-17T20:59:14.0124669Z Running tests... 2022-03-17T20:59:14.0125112Z ---------------------------------------------------------------------- 2022-03-17T20:59:15.5390261Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:15.5811691Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33440 2022-03-17T20:59:15.5937908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33441 2022-03-17T20:59:16.6756368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:16.6757279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:16.6758719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:16.6760068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:16.6766787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:16.6767716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:18.3116364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:18.3117147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:18.3132427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:18.3133400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:18.3291900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:18.3292947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:18.3306111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:18.3307089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:18.6016038Z ok (4.589s) 2022-03-17T20:59:18.6016448Z 2022-03-17T20:59:18.6017083Z ---------------------------------------------------------------------- 2022-03-17T20:59:18.6017662Z Ran 1 test in 4.589s 2022-03-17T20:59:18.6017971Z 2022-03-17T20:59:18.6018141Z OK 2022-03-17T20:59:18.6018394Z 2022-03-17T20:59:18.6018605Z Generating XML reports... 2022-03-17T20:59:18.6060893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205914.xml 2022-03-17T20:59:19.8908027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:19.8921581Z 2022-03-17T20:59:19.8922010Z Running tests... 2022-03-17T20:59:19.8922509Z ---------------------------------------------------------------------- 2022-03-17T20:59:21.4301951Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:21.4728212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33561 2022-03-17T20:59:21.4855672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33562 2022-03-17T20:59:22.5804788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:22.5805314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:22.5806113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:22.5806802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:22.5813313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:24.2123835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:24.2124366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:24.2124858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:24.2137111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:24.2137568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:24.2277531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:24.2278006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:24.2289018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:24.2289479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:24.5935333Z ok (4.701s) 2022-03-17T20:59:24.5935682Z 2022-03-17T20:59:24.5936199Z ---------------------------------------------------------------------- 2022-03-17T20:59:24.5936525Z Ran 1 test in 4.701s 2022-03-17T20:59:24.5936704Z 2022-03-17T20:59:24.5936775Z OK 2022-03-17T20:59:24.5936911Z 2022-03-17T20:59:24.5937042Z Generating XML reports... 2022-03-17T20:59:24.5980403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205919.xml 2022-03-17T20:59:25.8836591Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:25.8850947Z 2022-03-17T20:59:25.8851462Z Running tests... 2022-03-17T20:59:25.8851935Z ---------------------------------------------------------------------- 2022-03-17T20:59:27.3967987Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:27.4399976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33682 2022-03-17T20:59:27.4528271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33683 2022-03-17T20:59:28.5381468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:28.5381997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:28.5382852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:28.5383512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:28.5489714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:28.6396226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:29.9178031Z 2022-03-17T20:59:30.1606641Z ok (4.275s) 2022-03-17T20:59:30.1606992Z 2022-03-17T20:59:30.1607472Z ---------------------------------------------------------------------- 2022-03-17T20:59:30.1607739Z Ran 1 test in 4.276s 2022-03-17T20:59:30.1608191Z 2022-03-17T20:59:30.1608330Z OK 2022-03-17T20:59:30.1608473Z 2022-03-17T20:59:30.1608535Z Generating XML reports... 2022-03-17T20:59:30.1649603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205925.xml 2022-03-17T20:59:31.4622957Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:31.4637464Z 2022-03-17T20:59:31.4637698Z Running tests... 2022-03-17T20:59:31.4638170Z ---------------------------------------------------------------------- 2022-03-17T20:59:32.9954604Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:33.0380800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33799 2022-03-17T20:59:33.0509721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33800 2022-03-17T20:59:34.1485098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:34.1485666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:34.1486414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:34.1487107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:34.1595502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:34.2499703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:35.8583496Z ok (4.394s) 2022-03-17T20:59:35.8583945Z 2022-03-17T20:59:35.8584454Z ---------------------------------------------------------------------- 2022-03-17T20:59:35.8584845Z Ran 1 test in 4.395s 2022-03-17T20:59:35.8584992Z 2022-03-17T20:59:35.8585107Z OK 2022-03-17T20:59:35.8585248Z 2022-03-17T20:59:35.8585361Z Generating XML reports... 2022-03-17T20:59:35.8629263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205931.xml 2022-03-17T20:59:37.1632719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:37.1646593Z 2022-03-17T20:59:37.1646843Z Running tests... 2022-03-17T20:59:37.1647274Z ---------------------------------------------------------------------- 2022-03-17T20:59:38.7124779Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:38.7557124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33916 2022-03-17T20:59:38.7684186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33917 2022-03-17T20:59:39.8609046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:39.8609838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:39.8610852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:39.8611570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:39.8617668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:39.8618133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:41.4982719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:41.4983263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:41.7762437Z ok (4.611s) 2022-03-17T20:59:41.7762873Z 2022-03-17T20:59:41.7763873Z ---------------------------------------------------------------------- 2022-03-17T20:59:41.7764478Z Ran 1 test in 4.611s 2022-03-17T20:59:41.7764795Z 2022-03-17T20:59:41.7764937Z OK 2022-03-17T20:59:41.7765181Z 2022-03-17T20:59:41.7765414Z Generating XML reports... 2022-03-17T20:59:41.7807677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205937.xml 2022-03-17T20:59:43.0823488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:43.0837767Z 2022-03-17T20:59:43.0837886Z Running tests... 2022-03-17T20:59:43.0838628Z ---------------------------------------------------------------------- 2022-03-17T20:59:44.6318513Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:44.6750248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34037 2022-03-17T20:59:44.6879619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34038 2022-03-17T20:59:45.7828980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:45.7829538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:45.7830331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:45.7831028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:45.7937153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:45.8843641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:47.4520180Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:59:47.7960141Z ok (4.712s) 2022-03-17T20:59:47.7960372Z 2022-03-17T20:59:47.7960730Z ---------------------------------------------------------------------- 2022-03-17T20:59:47.7961054Z Ran 1 test in 4.712s 2022-03-17T20:59:47.7961211Z 2022-03-17T20:59:47.7961308Z OK 2022-03-17T20:59:47.7964297Z 2022-03-17T20:59:47.7964741Z Generating XML reports... 2022-03-17T20:59:47.8004376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205943.xml 2022-03-17T20:59:49.1026449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:49.1040418Z 2022-03-17T20:59:49.1040661Z Running tests... 2022-03-17T20:59:49.1041106Z ---------------------------------------------------------------------- 2022-03-17T20:59:50.6356815Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:50.6786383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34158 2022-03-17T20:59:50.6915587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34159 2022-03-17T20:59:51.7952717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:51.7953258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:51.7954054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:51.7954790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:51.7960581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:51.7961563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:53.4244554Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:59:53.4246708Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T20:59:53.7994840Z ok (4.695s) 2022-03-17T20:59:53.7995134Z 2022-03-17T20:59:53.7995713Z ---------------------------------------------------------------------- 2022-03-17T20:59:53.7996011Z Ran 1 test in 4.695s 2022-03-17T20:59:53.7996186Z 2022-03-17T20:59:53.7996283Z OK 2022-03-17T20:59:53.7996433Z 2022-03-17T20:59:53.7996564Z Generating XML reports... 2022-03-17T20:59:53.8039700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205949.xml 2022-03-17T20:59:55.0941659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:55.0955972Z 2022-03-17T20:59:55.0956222Z Running tests... 2022-03-17T20:59:55.0956644Z ---------------------------------------------------------------------- 2022-03-17T20:59:56.6152748Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T20:59:56.6581049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34279 2022-03-17T20:59:56.6708265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34280 2022-03-17T20:59:57.7905466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T20:59:57.7906002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T20:59:57.7906808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:57.7907511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T20:59:57.7915227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T20:59:57.7916421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T20:59:57.7942208Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7943715Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7948154Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:994.) 2022-03-17T20:59:57.7949504Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T20:59:57.7950937Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:994.) 2022-03-17T20:59:57.7951988Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T20:59:57.7952799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:57.7953414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T20:59:57.7956711Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7958217Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7962666Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7964128Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7968096Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7969591Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7973787Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7975501Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7979397Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.7981273Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T20:59:57.9755131Z ok (2.879s) 2022-03-17T20:59:57.9755591Z 2022-03-17T20:59:57.9756253Z ---------------------------------------------------------------------- 2022-03-17T20:59:57.9756868Z Ran 1 test in 2.880s 2022-03-17T20:59:57.9757141Z 2022-03-17T20:59:57.9757297Z OK 2022-03-17T20:59:57.9757550Z 2022-03-17T20:59:57.9757783Z Generating XML reports... 2022-03-17T20:59:57.9801839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205955.xml 2022-03-17T20:59:59.2552503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T20:59:59.2566816Z 2022-03-17T20:59:59.2567295Z Running tests... 2022-03-17T20:59:59.2567960Z ---------------------------------------------------------------------- 2022-03-17T21:00:00.7949035Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:00.8381338Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34398 2022-03-17T21:00:00.8510112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34399 2022-03-17T21:00:01.9391934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:01.9392650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:01.9393678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:01.9394418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:01.9499595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:02.0407183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:03.6269589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:03.6270190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:03.9588012Z ok (4.702s) 2022-03-17T21:00:03.9588456Z 2022-03-17T21:00:03.9588855Z ---------------------------------------------------------------------- 2022-03-17T21:00:03.9589287Z Ran 1 test in 4.702s 2022-03-17T21:00:03.9589556Z 2022-03-17T21:00:03.9589655Z OK 2022-03-17T21:00:03.9590031Z 2022-03-17T21:00:03.9590165Z Generating XML reports... 2022-03-17T21:00:03.9631893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205959.xml 2022-03-17T21:00:05.2646534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:00:05.2661012Z 2022-03-17T21:00:05.2661127Z Running tests... 2022-03-17T21:00:05.2662035Z ---------------------------------------------------------------------- 2022-03-17T21:00:06.8099864Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:06.8524934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34519 2022-03-17T21:00:06.8652745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34520 2022-03-17T21:00:07.9602145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:07.9602894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:07.9603738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:07.9604700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:07.9610872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:07.9611491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:09.3053711Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T21:00:09.3055543Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T21:00:09.3069151Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T21:00:09.3070084Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T21:00:09.8728914Z ok (4.606s) 2022-03-17T21:00:09.8729344Z 2022-03-17T21:00:09.8729907Z ---------------------------------------------------------------------- 2022-03-17T21:00:09.8730216Z Ran 1 test in 4.607s 2022-03-17T21:00:09.8730365Z 2022-03-17T21:00:09.8730460Z OK 2022-03-17T21:00:09.8731417Z 2022-03-17T21:00:09.8731654Z Generating XML reports... 2022-03-17T21:00:09.8772374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210005.xml 2022-03-17T21:00:11.1782528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:00:11.1797741Z 2022-03-17T21:00:11.1798073Z Running tests... 2022-03-17T21:00:11.1798521Z ---------------------------------------------------------------------- 2022-03-17T21:00:12.7097261Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:12.7524922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34670 2022-03-17T21:00:12.7651865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34671 2022-03-17T21:00:13.8789881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:13.8790430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:13.8791455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:13.8792151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:13.8897720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:13.9803947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:18.8869278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:18.8870575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:21.8845668Z ok (10.704s) 2022-03-17T21:00:21.8845967Z 2022-03-17T21:00:21.8846382Z ---------------------------------------------------------------------- 2022-03-17T21:00:21.8846712Z Ran 1 test in 10.705s 2022-03-17T21:00:21.8846879Z 2022-03-17T21:00:21.8846971Z OK 2022-03-17T21:00:21.8847113Z 2022-03-17T21:00:21.8847191Z Generating XML reports... 2022-03-17T21:00:21.8890363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210011.xml 2022-03-17T21:00:23.1831379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:00:23.1845237Z 2022-03-17T21:00:23.1845373Z Running tests... 2022-03-17T21:00:23.1846443Z ---------------------------------------------------------------------- 2022-03-17T21:00:24.7070385Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:24.7498467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34791 2022-03-17T21:00:24.7627203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34792 2022-03-17T21:00:25.8732415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:25.8733025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:25.8733804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:25.8734475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:25.8740874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:25.8741808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:27.4935902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:27.4936407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:27.8707390Z ok (4.686s) 2022-03-17T21:00:27.8707695Z 2022-03-17T21:00:27.8708291Z ---------------------------------------------------------------------- 2022-03-17T21:00:27.8708662Z Ran 1 test in 4.686s 2022-03-17T21:00:27.8708833Z 2022-03-17T21:00:27.8708936Z OK 2022-03-17T21:00:27.8709055Z 2022-03-17T21:00:27.8709196Z Generating XML reports... 2022-03-17T21:00:27.8751811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210023.xml 2022-03-17T21:00:29.1676984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:00:29.1690828Z 2022-03-17T21:00:29.1691101Z Running tests... 2022-03-17T21:00:29.1691578Z ---------------------------------------------------------------------- 2022-03-17T21:00:30.7113506Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:30.7543767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34912 2022-03-17T21:00:30.7673774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34913 2022-03-17T21:00:31.8332357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:31.8332893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:31.8333623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:31.8334313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:31.8341032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:31.8341511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:33.4510694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:33.4511234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:33.4526658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:33.4527098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:34.0756496Z ok (4.906s) 2022-03-17T21:00:34.0756725Z 2022-03-17T21:00:34.0757120Z ---------------------------------------------------------------------- 2022-03-17T21:00:34.0757462Z Ran 1 test in 4.907s 2022-03-17T21:00:34.0757632Z 2022-03-17T21:00:34.0757732Z OK 2022-03-17T21:00:34.0759079Z 2022-03-17T21:00:34.0760759Z Generating XML reports... 2022-03-17T21:00:34.0801238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210029.xml 2022-03-17T21:00:35.3771275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:00:35.3786197Z 2022-03-17T21:00:35.3786620Z Running tests... 2022-03-17T21:00:35.3787111Z ---------------------------------------------------------------------- 2022-03-17T21:00:36.9109052Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:36.9538210Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35033 2022-03-17T21:00:36.9667210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35034 2022-03-17T21:00:38.0682569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:38.0683109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:38.0683917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:38.0684599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:38.0691255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:38.0692097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:38.0800276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:00:38.0800751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:00:38.0801433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:00:38.0802181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:00:39.7337174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:39.7337711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:39.7352646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:39.7353130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:40.3762074Z ok (4.997s) 2022-03-17T21:00:40.3762259Z 2022-03-17T21:00:40.3762677Z ---------------------------------------------------------------------- 2022-03-17T21:00:40.3763016Z Ran 1 test in 4.998s 2022-03-17T21:00:40.3763181Z 2022-03-17T21:00:40.3763256Z OK 2022-03-17T21:00:40.3763391Z 2022-03-17T21:00:40.3763536Z Generating XML reports... 2022-03-17T21:00:40.3806257Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210035.xml 2022-03-17T21:00:41.6894628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:00:41.6909966Z 2022-03-17T21:00:41.6910250Z Running tests... 2022-03-17T21:00:41.6910703Z ---------------------------------------------------------------------- 2022-03-17T21:00:43.2177717Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:43.2624026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35160 2022-03-17T21:00:43.2753854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35161 2022-03-17T21:00:44.3490204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:44.3491358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:44.3492177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:44.3492852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:44.3498690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:44.3499705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:44.3500765Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-03-17T21:00:44.3501497Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-03-17T21:00:45.9701132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:45.9701939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:45.9716742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:45.9717750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:45.9960990Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-03-17T21:00:45.9961988Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-03-17T21:00:46.2481611Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-03-17T21:00:46.2482569Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-03-17T21:00:46.2554777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:46.2555760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:46.2571769Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:46.2572753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:46.8840042Z ok (5.193s) 2022-03-17T21:00:46.8840371Z 2022-03-17T21:00:46.8841162Z ---------------------------------------------------------------------- 2022-03-17T21:00:46.8841733Z Ran 1 test in 5.193s 2022-03-17T21:00:46.8841907Z 2022-03-17T21:00:46.8841980Z OK 2022-03-17T21:00:46.8842118Z 2022-03-17T21:00:46.8842987Z Generating XML reports... 2022-03-17T21:00:46.8883919Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210041.xml 2022-03-17T21:00:48.1853873Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:00:48.1868830Z 2022-03-17T21:00:48.1869083Z Running tests... 2022-03-17T21:00:48.1869866Z ---------------------------------------------------------------------- 2022-03-17T21:00:49.7235262Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:49.7666476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35281 2022-03-17T21:00:49.7800815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35282 2022-03-17T21:00:50.8876863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:50.8877599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:50.8878589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:50.8879334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:50.8985118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:50.8987669Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:00:50.9891174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:50.9892488Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:00:52.5787194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:52.5787758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:52.5803517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:52.5803996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:52.5810527Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T21:00:52.5811113Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T21:00:52.5842550Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T21:00:52.5843481Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T21:00:52.5844345Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T21:00:52.5845239Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T21:00:52.5846168Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-03-17T21:00:52.5847119Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-03-17T21:00:52.9157617Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:00:52.9158726Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:00:52.9234157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:52.9234674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:52.9251083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:52.9251627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:52.9256696Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T21:00:52.9257242Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T21:00:52.9289029Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T21:00:52.9289943Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T21:00:52.9290608Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T21:00:52.9291266Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T21:00:53.5890132Z ok (5.402s) 2022-03-17T21:00:53.5890351Z 2022-03-17T21:00:53.5890910Z ---------------------------------------------------------------------- 2022-03-17T21:00:53.5891332Z Ran 1 test in 5.402s 2022-03-17T21:00:53.5891480Z 2022-03-17T21:00:53.5891576Z OK 2022-03-17T21:00:53.5891720Z 2022-03-17T21:00:53.5891855Z Generating XML reports... 2022-03-17T21:00:53.5934181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210048.xml 2022-03-17T21:00:54.8863377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:00:54.8878127Z 2022-03-17T21:00:54.8878387Z Running tests... 2022-03-17T21:00:54.8878841Z ---------------------------------------------------------------------- 2022-03-17T21:00:56.4360449Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:00:56.4804243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35402 2022-03-17T21:00:56.4933827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35403 2022-03-17T21:00:57.5506931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:00:57.5507510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:00:57.5508275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:57.5508964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:00:57.5615094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:00:57.6521373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:00:59.0013434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6c7cjog 2022-03-17T21:00:59.0014037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp169oq37a 2022-03-17T21:00:59.0014541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6c7cjog/_remote_module_non_sriptable.py 2022-03-17T21:00:59.0015283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp169oq37a/_remote_module_non_sriptable.py 2022-03-17T21:00:59.3196367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:59.3196884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:59.3868814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:59.3869387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:00:59.8016943Z ok (4.914s) 2022-03-17T21:00:59.8017303Z 2022-03-17T21:00:59.8017814Z ---------------------------------------------------------------------- 2022-03-17T21:00:59.8018174Z Ran 1 test in 4.914s 2022-03-17T21:00:59.8018346Z 2022-03-17T21:00:59.8018436Z OK 2022-03-17T21:00:59.8018578Z 2022-03-17T21:00:59.8018769Z Generating XML reports... 2022-03-17T21:00:59.8070349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210054.xml 2022-03-17T21:01:01.1082754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:01.1097232Z 2022-03-17T21:01:01.1097544Z Running tests... 2022-03-17T21:01:01.1098000Z ---------------------------------------------------------------------- 2022-03-17T21:01:02.6562996Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:02.6992914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35553 2022-03-17T21:01:02.7123861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35554 2022-03-17T21:01:03.8341983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:03.8342985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:03.8344304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:03.8345683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:03.8352295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:03.8353055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:05.2318005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot3bojha 2022-03-17T21:01:05.2319013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpippf3wf5 2022-03-17T21:01:05.2320029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot3bojha/_remote_module_non_sriptable.py 2022-03-17T21:01:05.2321098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpippf3wf5/_remote_module_non_sriptable.py 2022-03-17T21:01:05.5422792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:05.5423283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:05.6085243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:05.6086077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:06.0205675Z ok (4.911s) 2022-03-17T21:01:06.0206009Z 2022-03-17T21:01:06.0206455Z ---------------------------------------------------------------------- 2022-03-17T21:01:06.0206775Z Ran 1 test in 4.911s 2022-03-17T21:01:06.0206943Z 2022-03-17T21:01:06.0207002Z OK 2022-03-17T21:01:06.0207128Z 2022-03-17T21:01:06.0207260Z Generating XML reports... 2022-03-17T21:01:06.0249604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210101.xml 2022-03-17T21:01:07.3128094Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:07.3141350Z 2022-03-17T21:01:07.3141602Z Running tests... 2022-03-17T21:01:07.3142292Z ---------------------------------------------------------------------- 2022-03-17T21:01:08.8414896Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:08.8854069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35704 2022-03-17T21:01:08.8988444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35705 2022-03-17T21:01:09.9805459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:09.9806354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:09.9807165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:09.9807866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:09.9913161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:10.0821560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:11.4513832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7g419_43 2022-03-17T21:01:11.4514417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1m070d8f 2022-03-17T21:01:11.4514963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1m070d8f/_remote_module_non_sriptable.py 2022-03-17T21:01:11.4515478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7g419_43/_remote_module_non_sriptable.py 2022-03-17T21:01:11.7637536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:11.7638068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:11.8307254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:11.8307824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:12.2072334Z ok (4.893s) 2022-03-17T21:01:12.2072549Z 2022-03-17T21:01:12.2072928Z ---------------------------------------------------------------------- 2022-03-17T21:01:12.2073246Z Ran 1 test in 4.893s 2022-03-17T21:01:12.2073419Z 2022-03-17T21:01:12.2073514Z OK 2022-03-17T21:01:12.2073649Z 2022-03-17T21:01:12.2073778Z Generating XML reports... 2022-03-17T21:01:12.2116343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210107.xml 2022-03-17T21:01:13.5187711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:13.5202994Z 2022-03-17T21:01:13.5203541Z Running tests... 2022-03-17T21:01:13.5204024Z ---------------------------------------------------------------------- 2022-03-17T21:01:15.0661073Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:15.1104721Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35855 2022-03-17T21:01:15.1234669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35856 2022-03-17T21:01:16.2068494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:16.2068980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:16.2069987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:16.2070710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:16.2177142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:16.3082155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:17.6569393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d97l0mv 2022-03-17T21:01:17.6570168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2t0likj 2022-03-17T21:01:17.6570657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d97l0mv/_remote_module_non_sriptable.py 2022-03-17T21:01:17.6571196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2t0likj/_remote_module_non_sriptable.py 2022-03-17T21:01:17.9712965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:17.9713480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:18.0442903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:18.0443409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:18.4317256Z ok (4.911s) 2022-03-17T21:01:18.4317737Z 2022-03-17T21:01:18.4318438Z ---------------------------------------------------------------------- 2022-03-17T21:01:18.4318801Z Ran 1 test in 4.911s 2022-03-17T21:01:18.4318946Z 2022-03-17T21:01:18.4319040Z OK 2022-03-17T21:01:18.4319177Z 2022-03-17T21:01:18.4319312Z Generating XML reports... 2022-03-17T21:01:18.4361940Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210113.xml 2022-03-17T21:01:19.7292544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:19.7308061Z 2022-03-17T21:01:19.7308615Z Running tests... 2022-03-17T21:01:19.7309249Z ---------------------------------------------------------------------- 2022-03-17T21:01:21.2731206Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:21.3175942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36006 2022-03-17T21:01:21.3305564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36007 2022-03-17T21:01:22.4379585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:22.4380148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:22.4381212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:22.4381933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:22.4489660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:22.5393544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:23.8723185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvf43g4qo 2022-03-17T21:01:23.8724336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbuw7typ 2022-03-17T21:01:23.8724894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvf43g4qo/_remote_module_non_sriptable.py 2022-03-17T21:01:23.8725428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbuw7typ/_remote_module_non_sriptable.py 2022-03-17T21:01:24.1978031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:24.1979261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:24.2689924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:24.2690630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:24.6387525Z ok (4.908s) 2022-03-17T21:01:24.6387884Z 2022-03-17T21:01:24.6388336Z ---------------------------------------------------------------------- 2022-03-17T21:01:24.6388998Z Ran 1 test in 4.908s 2022-03-17T21:01:24.6389396Z 2022-03-17T21:01:24.6389551Z OK 2022-03-17T21:01:24.6389787Z 2022-03-17T21:01:24.6389933Z Generating XML reports... 2022-03-17T21:01:24.6431347Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210119.xml 2022-03-17T21:01:25.8967608Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:25.8983080Z 2022-03-17T21:01:25.8983224Z Running tests... 2022-03-17T21:01:25.8984755Z ---------------------------------------------------------------------- 2022-03-17T21:01:27.4346597Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:27.4776763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36157 2022-03-17T21:01:27.4908113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36158 2022-03-17T21:01:28.5894605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:28.5895158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:28.5895946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:28.5896660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:28.6003712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:28.6909676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:30.0235967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0iigldm 2022-03-17T21:01:30.0237124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvs48a3eu 2022-03-17T21:01:30.0238179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0iigldm/_remote_module_non_sriptable.py 2022-03-17T21:01:30.0239232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvs48a3eu/_remote_module_non_sriptable.py 2022-03-17T21:01:30.3474326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:30.3474855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:30.4160244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:30.4160766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:30.7992570Z ok (4.901s) 2022-03-17T21:01:30.7992849Z 2022-03-17T21:01:30.7994129Z ---------------------------------------------------------------------- 2022-03-17T21:01:30.7994766Z Ran 1 test in 4.901s 2022-03-17T21:01:30.7995011Z 2022-03-17T21:01:30.7995176Z OK 2022-03-17T21:01:30.7995463Z 2022-03-17T21:01:30.7995711Z Generating XML reports... 2022-03-17T21:01:30.8038896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210125.xml 2022-03-17T21:01:32.1073214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:32.1086958Z 2022-03-17T21:01:32.1087392Z Running tests... 2022-03-17T21:01:32.1088151Z ---------------------------------------------------------------------- 2022-03-17T21:01:33.6474287Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:33.6909054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36308 2022-03-17T21:01:33.7038171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36309 2022-03-17T21:01:34.7579627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:34.7581031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:34.7582396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:34.7583118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:34.7687235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:34.8595429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:36.1954483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo4gk9ie 2022-03-17T21:01:36.1955628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9c7o8jte 2022-03-17T21:01:36.1956601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo4gk9ie/_remote_module_non_sriptable.py 2022-03-17T21:01:36.1957613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9c7o8jte/_remote_module_non_sriptable.py 2022-03-17T21:01:36.5070193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:36.5070679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:36.5715639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:36.5716128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:37.0120565Z ok (4.903s) 2022-03-17T21:01:37.0120778Z 2022-03-17T21:01:37.0121166Z ---------------------------------------------------------------------- 2022-03-17T21:01:37.0121495Z Ran 1 test in 4.903s 2022-03-17T21:01:37.0121664Z 2022-03-17T21:01:37.0121757Z OK 2022-03-17T21:01:37.0132477Z 2022-03-17T21:01:37.0132645Z Generating XML reports... 2022-03-17T21:01:37.0166169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210132.xml 2022-03-17T21:01:38.3076942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:38.3092507Z 2022-03-17T21:01:38.3093063Z Running tests... 2022-03-17T21:01:38.3093581Z ---------------------------------------------------------------------- 2022-03-17T21:01:39.8479176Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:39.8906227Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36459 2022-03-17T21:01:39.9031931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36460 2022-03-17T21:01:41.0298550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:41.0299087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:41.0299882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:41.0300890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:41.0307994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:41.0308494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:42.4287222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyofu4j_7 2022-03-17T21:01:42.4287805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps27buffs 2022-03-17T21:01:42.4289194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyofu4j_7/_remote_module_non_sriptable.py 2022-03-17T21:01:42.4289776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps27buffs/_remote_module_non_sriptable.py 2022-03-17T21:01:42.7650555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:42.7651058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:42.8338738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:42.8339272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:43.2116649Z ok (4.902s) 2022-03-17T21:01:43.2116866Z 2022-03-17T21:01:43.2117258Z ---------------------------------------------------------------------- 2022-03-17T21:01:43.2117615Z Ran 1 test in 4.902s 2022-03-17T21:01:43.2117779Z 2022-03-17T21:01:43.2117872Z OK 2022-03-17T21:01:43.2118003Z 2022-03-17T21:01:43.2118130Z Generating XML reports... 2022-03-17T21:01:43.2161774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210138.xml 2022-03-17T21:01:44.4756377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:44.4770422Z 2022-03-17T21:01:44.4770812Z Running tests... 2022-03-17T21:01:44.4771312Z ---------------------------------------------------------------------- 2022-03-17T21:01:46.0360270Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:46.0792534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36610 2022-03-17T21:01:46.0923394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36611 2022-03-17T21:01:47.1784379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:47.1784967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:47.1785679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:47.1786348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:47.1892744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:47.2800248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:48.6353077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphe1pz3rc 2022-03-17T21:01:48.6353944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepz545wh 2022-03-17T21:01:48.6354465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphe1pz3rc/_remote_module_non_sriptable.py 2022-03-17T21:01:48.6355618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepz545wh/_remote_module_non_sriptable.py 2022-03-17T21:01:48.9619975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:48.9620826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:49.0310608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:49.0311139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:49.4005053Z ok (4.923s) 2022-03-17T21:01:49.4005516Z 2022-03-17T21:01:49.4006068Z ---------------------------------------------------------------------- 2022-03-17T21:01:49.4006436Z Ran 1 test in 4.923s 2022-03-17T21:01:49.4006576Z 2022-03-17T21:01:49.4006673Z OK 2022-03-17T21:01:49.4006754Z 2022-03-17T21:01:49.4006888Z Generating XML reports... 2022-03-17T21:01:49.4049026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210144.xml 2022-03-17T21:01:50.6971097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:50.6985249Z 2022-03-17T21:01:50.6985510Z Running tests... 2022-03-17T21:01:50.6986064Z ---------------------------------------------------------------------- 2022-03-17T21:01:52.2178279Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:52.2606721Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36761 2022-03-17T21:01:52.2733980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36762 2022-03-17T21:01:53.3692439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:53.3692990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:53.3693684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:53.3694376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:53.3700931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:01:53.3701800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:54.7556716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgou2ikzq 2022-03-17T21:01:54.7557337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6usu5qwb 2022-03-17T21:01:54.7557792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgou2ikzq/_remote_module_non_sriptable.py 2022-03-17T21:01:54.7558313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6usu5qwb/_remote_module_non_sriptable.py 2022-03-17T21:01:55.0781288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:55.0781797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:55.1451666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:55.1452196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:01:55.4813755Z ok (4.782s) 2022-03-17T21:01:55.4813962Z 2022-03-17T21:01:55.4814321Z ---------------------------------------------------------------------- 2022-03-17T21:01:55.4814831Z Ran 1 test in 4.783s 2022-03-17T21:01:55.4815366Z 2022-03-17T21:01:55.4815446Z OK 2022-03-17T21:01:55.4815574Z 2022-03-17T21:01:55.4815709Z Generating XML reports... 2022-03-17T21:01:55.4856832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210150.xml 2022-03-17T21:01:56.7847079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:01:56.7861375Z 2022-03-17T21:01:56.7861604Z Running tests... 2022-03-17T21:01:56.7862047Z ---------------------------------------------------------------------- 2022-03-17T21:01:58.3270729Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:01:58.3692619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36912 2022-03-17T21:01:58.3819492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36913 2022-03-17T21:01:59.4859667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:01:59.4860187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:01:59.4861194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:59.4861891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:01:59.4967779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:01:59.5874057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:00.9496822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimyh9dfp 2022-03-17T21:02:00.9497393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmhucx2h 2022-03-17T21:02:00.9498313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimyh9dfp/_remote_module_non_sriptable.py 2022-03-17T21:02:00.9498860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmhucx2h/_remote_module_non_sriptable.py 2022-03-17T21:02:01.2723942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:01.2724512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:01.3457906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:01.3458408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:01.7903899Z ok (5.004s) 2022-03-17T21:02:01.7904469Z 2022-03-17T21:02:01.7905311Z ---------------------------------------------------------------------- 2022-03-17T21:02:01.7905793Z Ran 1 test in 5.004s 2022-03-17T21:02:01.7905945Z 2022-03-17T21:02:01.7906068Z OK 2022-03-17T21:02:01.7906236Z 2022-03-17T21:02:01.7906371Z Generating XML reports... 2022-03-17T21:02:01.7948980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210156.xml 2022-03-17T21:02:03.1011342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:03.1025468Z 2022-03-17T21:02:03.1025700Z Running tests... 2022-03-17T21:02:03.1026125Z ---------------------------------------------------------------------- 2022-03-17T21:02:04.6467987Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:02:04.6896859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37063 2022-03-17T21:02:04.7026587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37064 2022-03-17T21:02:05.8050800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:05.8051572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:05.8052335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:05.8053025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:05.8161233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:05.9064853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:07.2571892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3i5h17p6 2022-03-17T21:02:07.2572488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpcg3mg3 2022-03-17T21:02:07.2573375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3i5h17p6/_remote_module_non_sriptable.py 2022-03-17T21:02:07.2574016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpcg3mg3/_remote_module_non_sriptable.py 2022-03-17T21:02:07.5698307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:07.5698782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:07.6368921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:07.6369427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:08.0122834Z ok (4.909s) 2022-03-17T21:02:08.0123002Z 2022-03-17T21:02:08.0123669Z ---------------------------------------------------------------------- 2022-03-17T21:02:08.0124140Z Ran 1 test in 4.910s 2022-03-17T21:02:08.0124223Z 2022-03-17T21:02:08.0124316Z OK 2022-03-17T21:02:08.0124449Z 2022-03-17T21:02:08.0124577Z Generating XML reports... 2022-03-17T21:02:08.0166630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210203.xml 2022-03-17T21:02:09.3098871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:09.3113430Z 2022-03-17T21:02:09.3113815Z Running tests... 2022-03-17T21:02:09.3114292Z ---------------------------------------------------------------------- 2022-03-17T21:02:10.8298522Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:02:10.8722402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37214 2022-03-17T21:02:10.8848518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37215 2022-03-17T21:02:11.9832172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:11.9832834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:11.9833636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:11.9834327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:11.9840926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:11.9841537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:13.5920376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.5920932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.6074428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.6074939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.6203777Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:02:13.6205439Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:02:13.6567564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.6568053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.6721111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.6721777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.6825660Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:02:13.6826530Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:02:13.6827679Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:02:13.6828497Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:02:13.6879907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.6880901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.7033508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.7034582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:13.9927742Z ok (4.681s) 2022-03-17T21:02:13.9928133Z 2022-03-17T21:02:13.9928720Z ---------------------------------------------------------------------- 2022-03-17T21:02:13.9929023Z Ran 1 test in 4.681s 2022-03-17T21:02:13.9929331Z 2022-03-17T21:02:13.9929496Z OK 2022-03-17T21:02:13.9929765Z 2022-03-17T21:02:13.9930046Z Generating XML reports... 2022-03-17T21:02:13.9972584Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210209.xml 2022-03-17T21:02:15.2975315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:15.2989430Z 2022-03-17T21:02:15.2990217Z Running tests... 2022-03-17T21:02:15.2991160Z ---------------------------------------------------------------------- 2022-03-17T21:02:16.8443764Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:02:16.8873006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37335 2022-03-17T21:02:16.9000153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37336 2022-03-17T21:02:17.9994551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:17.9995095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:17.9995889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:17.9996563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:18.0103178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:18.1009085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:20.5087150Z ok (5.209s) 2022-03-17T21:02:20.5092102Z 2022-03-17T21:02:20.5092756Z ---------------------------------------------------------------------- 2022-03-17T21:02:20.5093115Z Ran 1 test in 5.210s 2022-03-17T21:02:20.5093279Z 2022-03-17T21:02:20.5093373Z OK 2022-03-17T21:02:20.5093510Z 2022-03-17T21:02:20.5093626Z Generating XML reports... 2022-03-17T21:02:20.5145233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210215.xml 2022-03-17T21:02:21.8020795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:21.8034696Z 2022-03-17T21:02:21.8035238Z Running tests... 2022-03-17T21:02:21.8035711Z ---------------------------------------------------------------------- 2022-03-17T21:02:23.3198885Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:02:23.3621221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37452 2022-03-17T21:02:23.3748058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37453 2022-03-17T21:02:24.4408613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:24.4409149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:24.4409937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:24.4410628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:24.4518886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:24.5423005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:26.1009674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:26.1010235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:26.3825595Z ok (4.579s) 2022-03-17T21:02:26.3825815Z 2022-03-17T21:02:26.3826187Z ---------------------------------------------------------------------- 2022-03-17T21:02:26.3826516Z Ran 1 test in 4.579s 2022-03-17T21:02:26.3826658Z 2022-03-17T21:02:26.3826782Z OK 2022-03-17T21:02:26.3826884Z 2022-03-17T21:02:26.3827002Z Generating XML reports... 2022-03-17T21:02:26.3869425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210221.xml 2022-03-17T21:02:27.6910215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:27.6924349Z 2022-03-17T21:02:27.6924732Z Running tests... 2022-03-17T21:02:27.6925245Z ---------------------------------------------------------------------- 2022-03-17T21:02:29.2236459Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:02:29.2664735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37573 2022-03-17T21:02:29.2793522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37574 2022-03-17T21:02:30.3888078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:30.3888645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:30.3889417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:30.3890371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:30.3895993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:30.3896477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:30.3951658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:30.3952154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:30.6841740Z ok (2.991s) 2022-03-17T21:02:30.6842035Z 2022-03-17T21:02:30.6842451Z ---------------------------------------------------------------------- 2022-03-17T21:02:30.6842775Z Ran 1 test in 2.992s 2022-03-17T21:02:30.6842940Z 2022-03-17T21:02:30.6843013Z OK 2022-03-17T21:02:30.6843153Z 2022-03-17T21:02:30.6843287Z Generating XML reports... 2022-03-17T21:02:30.6886298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210227.xml 2022-03-17T21:02:31.9720406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:31.9734314Z 2022-03-17T21:02:31.9734436Z Running tests... 2022-03-17T21:02:31.9735212Z ---------------------------------------------------------------------- 2022-03-17T21:02:33.5323370Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:02:33.5754302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37722 2022-03-17T21:02:33.5884764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37723 2022-03-17T21:02:34.6408992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:34.6411854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:34.6412685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:34.6513587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:34.6521619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:34.7427792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:36.3082584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:36.3083603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:02:36.6964970Z ok (4.723s) 2022-03-17T21:02:36.6965178Z 2022-03-17T21:02:36.6965575Z ---------------------------------------------------------------------- 2022-03-17T21:02:36.6965922Z Ran 1 test in 4.723s 2022-03-17T21:02:36.6966064Z 2022-03-17T21:02:36.6966153Z OK 2022-03-17T21:02:36.6966304Z 2022-03-17T21:02:36.6966441Z Generating XML reports... 2022-03-17T21:02:36.7010273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210231.xml 2022-03-17T21:02:38.0164950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:38.0179754Z 2022-03-17T21:02:38.0179926Z Running tests... 2022-03-17T21:02:38.0180718Z ---------------------------------------------------------------------- 2022-03-17T21:02:39.5663053Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:02:39.6096534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37843 2022-03-17T21:02:39.6225581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37844 2022-03-17T21:02:40.7627916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:40.7628633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:40.7629420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:40.7630122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:40.7636107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:40.7636792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:40.7744329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:02:40.7744934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:02:40.7745663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:02:40.7746358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:02:40.7952693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:02:40.7953172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:02:40.7953846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:02:40.7954525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:02:42.4306935Z ok (4.412s) 2022-03-17T21:02:42.4307161Z 2022-03-17T21:02:42.4307566Z ---------------------------------------------------------------------- 2022-03-17T21:02:42.4307921Z Ran 1 test in 4.413s 2022-03-17T21:02:42.4308066Z 2022-03-17T21:02:42.4308178Z OK 2022-03-17T21:02:42.4308309Z 2022-03-17T21:02:42.4308437Z Generating XML reports... 2022-03-17T21:02:42.4350470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210238.xml 2022-03-17T21:02:43.7355692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:43.7369754Z 2022-03-17T21:02:43.7370274Z Running tests... 2022-03-17T21:02:43.7370785Z ---------------------------------------------------------------------- 2022-03-17T21:02:45.2804889Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:02:45.3235354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37972 2022-03-17T21:02:45.3362834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37973 2022-03-17T21:02:46.4344419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:02:46.4344972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:02:46.4345716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:46.4346653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:02:46.4455059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:02:46.5356297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:02:46.5470673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:02:46.5471183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:02:46.5472004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:02:46.5472735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:02:46.5681216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:02:46.5681693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:02:46.5682392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:02:46.5683043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:02:58.2619591Z ok (14.525s) 2022-03-17T21:02:58.2619894Z 2022-03-17T21:02:58.2620526Z ---------------------------------------------------------------------- 2022-03-17T21:02:58.2620891Z Ran 1 test in 14.525s 2022-03-17T21:02:58.2621041Z 2022-03-17T21:02:58.2621133Z OK 2022-03-17T21:02:58.2621267Z 2022-03-17T21:02:58.2623752Z Generating XML reports... 2022-03-17T21:02:58.2664332Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210243.xml 2022-03-17T21:02:59.5651093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:02:59.5665634Z 2022-03-17T21:02:59.5665916Z Running tests... 2022-03-17T21:02:59.5666359Z ---------------------------------------------------------------------- 2022-03-17T21:03:01.1159163Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:01.1588397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38101 2022-03-17T21:03:01.1718676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38102 2022-03-17T21:03:02.2607238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:02.2607783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:02.2608570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:02.2609270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:02.2615900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:02.2616626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:04.6810123Z ok (5.114s) 2022-03-17T21:03:04.6810345Z 2022-03-17T21:03:04.6810700Z ---------------------------------------------------------------------- 2022-03-17T21:03:04.6811058Z Ran 1 test in 5.114s 2022-03-17T21:03:04.6811204Z 2022-03-17T21:03:04.6811317Z OK 2022-03-17T21:03:04.6811457Z 2022-03-17T21:03:04.6811599Z Generating XML reports... 2022-03-17T21:03:04.6856553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210259.xml 2022-03-17T21:03:05.9847108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:05.9861076Z 2022-03-17T21:03:05.9861378Z Running tests... 2022-03-17T21:03:05.9861807Z ---------------------------------------------------------------------- 2022-03-17T21:03:07.5167451Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:07.5594148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38222 2022-03-17T21:03:07.5722053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38223 2022-03-17T21:03:08.6524417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:08.6524997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:08.6525696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:08.6526386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:08.6532329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:08.6532806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:11.0806195Z ok (5.094s) 2022-03-17T21:03:11.0806499Z 2022-03-17T21:03:11.0807015Z ---------------------------------------------------------------------- 2022-03-17T21:03:11.0807554Z Ran 1 test in 5.094s 2022-03-17T21:03:11.0807717Z 2022-03-17T21:03:11.0807881Z OK 2022-03-17T21:03:11.0808140Z 2022-03-17T21:03:11.0808311Z Generating XML reports... 2022-03-17T21:03:11.0850869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210305.xml 2022-03-17T21:03:12.3834564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:12.3848249Z 2022-03-17T21:03:12.3848537Z Running tests... 2022-03-17T21:03:12.3848983Z ---------------------------------------------------------------------- 2022-03-17T21:03:13.9288963Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:13.9719361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38343 2022-03-17T21:03:13.9849478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38344 2022-03-17T21:03:15.1028612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:15.1029173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:15.1029935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:15.1030636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:15.1036794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:15.1037260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:17.0941362Z ok (4.709s) 2022-03-17T21:03:17.0941684Z 2022-03-17T21:03:17.0942088Z ---------------------------------------------------------------------- 2022-03-17T21:03:17.0942402Z Ran 1 test in 4.709s 2022-03-17T21:03:17.0942572Z 2022-03-17T21:03:17.0942668Z OK 2022-03-17T21:03:17.0942807Z 2022-03-17T21:03:17.0942944Z Generating XML reports... 2022-03-17T21:03:17.0987472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210312.xml 2022-03-17T21:03:18.3983149Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:18.3997499Z 2022-03-17T21:03:18.3997844Z Running tests... 2022-03-17T21:03:18.3998317Z ---------------------------------------------------------------------- 2022-03-17T21:03:19.9422634Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:19.9852878Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38460 2022-03-17T21:03:19.9979732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38461 2022-03-17T21:03:21.0865853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:21.0866809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:21.0867594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:21.0868290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:21.0873709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:21.0874634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:22.6859162Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:03:22.6939559Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:03:22.7021889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:22.7022402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:23.0056736Z ok (4.606s) 2022-03-17T21:03:23.0056892Z 2022-03-17T21:03:23.0057340Z ---------------------------------------------------------------------- 2022-03-17T21:03:23.0057893Z Ran 1 test in 4.606s 2022-03-17T21:03:23.0058059Z 2022-03-17T21:03:23.0058132Z OK 2022-03-17T21:03:23.0058267Z 2022-03-17T21:03:23.0058391Z Generating XML reports... 2022-03-17T21:03:23.0100828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210318.xml 2022-03-17T21:03:24.3072099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:24.3086074Z 2022-03-17T21:03:24.3086541Z Running tests... 2022-03-17T21:03:24.3087007Z ---------------------------------------------------------------------- 2022-03-17T21:03:25.8469511Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:25.8900143Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38581 2022-03-17T21:03:25.9027604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38582 2022-03-17T21:03:26.9890825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:26.9891447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:26.9892485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:26.9893158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:26.9899576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:26.9900464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:28.3289419Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:03:28.3290390Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:03:28.3291590Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:03:28.3292413Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:03:28.9103726Z ok (4.601s) 2022-03-17T21:03:28.9103955Z 2022-03-17T21:03:28.9104336Z ---------------------------------------------------------------------- 2022-03-17T21:03:28.9104686Z Ran 1 test in 4.602s 2022-03-17T21:03:28.9104835Z 2022-03-17T21:03:28.9107652Z OK 2022-03-17T21:03:28.9107933Z 2022-03-17T21:03:28.9108422Z Generating XML reports... 2022-03-17T21:03:28.9147612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210324.xml 2022-03-17T21:03:30.2207927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:30.2222191Z 2022-03-17T21:03:30.2222631Z Running tests... 2022-03-17T21:03:30.2223153Z ---------------------------------------------------------------------- 2022-03-17T21:03:31.7753008Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:31.8188084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38702 2022-03-17T21:03:31.8323195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38703 2022-03-17T21:03:32.9431408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:32.9431974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:32.9432744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:32.9433452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:32.9539249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:33.0445105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:35.0208130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:35.0208649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:35.0529306Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:03:35.0531297Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:03:35.4409153Z ok (5.218s) 2022-03-17T21:03:35.4409445Z 2022-03-17T21:03:35.4409866Z ---------------------------------------------------------------------- 2022-03-17T21:03:35.4410228Z Ran 1 test in 5.219s 2022-03-17T21:03:35.4410375Z 2022-03-17T21:03:35.4410465Z OK 2022-03-17T21:03:35.4410598Z 2022-03-17T21:03:35.4410730Z Generating XML reports... 2022-03-17T21:03:35.4452472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210330.xml 2022-03-17T21:03:36.7636224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:36.7650694Z 2022-03-17T21:03:36.7651106Z Running tests... 2022-03-17T21:03:36.7651592Z ---------------------------------------------------------------------- 2022-03-17T21:03:38.3224747Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:38.3655302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38827 2022-03-17T21:03:38.3789459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38828 2022-03-17T21:03:39.4715037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:39.4715613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:39.4716359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:39.4717032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:39.4723405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:39.4724111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:41.5386298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:41.5386842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:41.5748097Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:03:41.5749573Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:03:41.8876097Z ok (5.122s) 2022-03-17T21:03:41.8876277Z 2022-03-17T21:03:41.8877065Z ---------------------------------------------------------------------- 2022-03-17T21:03:41.8877426Z Ran 1 test in 5.122s 2022-03-17T21:03:41.8877595Z 2022-03-17T21:03:41.8877667Z OK 2022-03-17T21:03:41.8877819Z 2022-03-17T21:03:41.8877944Z Generating XML reports... 2022-03-17T21:03:41.8920213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210336.xml 2022-03-17T21:03:43.1848914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:43.1863364Z 2022-03-17T21:03:43.1864090Z Running tests... 2022-03-17T21:03:43.1864563Z ---------------------------------------------------------------------- 2022-03-17T21:03:44.7179779Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:44.7608664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38952 2022-03-17T21:03:44.7738456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38953 2022-03-17T21:03:45.8803038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:45.8803581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:45.8804324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:45.8805047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:45.8910697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:45.9817672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:47.5811011Z ok (4.394s) 2022-03-17T21:03:47.5811242Z 2022-03-17T21:03:47.5811595Z ---------------------------------------------------------------------- 2022-03-17T21:03:47.5812138Z Ran 1 test in 4.395s 2022-03-17T21:03:47.5812399Z 2022-03-17T21:03:47.5812493Z OK 2022-03-17T21:03:47.5812614Z 2022-03-17T21:03:47.5812712Z Generating XML reports... 2022-03-17T21:03:47.5854645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210343.xml 2022-03-17T21:03:48.8862806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:48.8880530Z 2022-03-17T21:03:48.8881476Z Running tests... 2022-03-17T21:03:48.8882124Z ---------------------------------------------------------------------- 2022-03-17T21:03:50.4115320Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:50.4553876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39069 2022-03-17T21:03:50.4687141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39070 2022-03-17T21:03:51.5259121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:51.5259764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:51.5260747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:51.5261441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:51.5268720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:51.5269218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:52.8588935Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:03:52.8590107Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:03:52.8591392Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:03:52.8592243Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:03:53.1474084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:53.1474690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:53.1541700Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:03:53.1543270Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:03:53.1679410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:53.1679909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:53.1760687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:53.1761511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:53.4764175Z ok (4.588s) 2022-03-17T21:03:53.4764453Z 2022-03-17T21:03:53.4764924Z ---------------------------------------------------------------------- 2022-03-17T21:03:53.4765482Z Ran 1 test in 4.589s 2022-03-17T21:03:53.4765655Z 2022-03-17T21:03:53.4765749Z OK 2022-03-17T21:03:53.4765878Z 2022-03-17T21:03:53.4766008Z Generating XML reports... 2022-03-17T21:03:53.4808440Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210348.xml 2022-03-17T21:03:54.7627337Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:03:54.7641374Z 2022-03-17T21:03:54.7641730Z Running tests... 2022-03-17T21:03:54.7642196Z ---------------------------------------------------------------------- 2022-03-17T21:03:56.2835986Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:03:56.3263392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39190 2022-03-17T21:03:56.3390813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39191 2022-03-17T21:03:57.4258354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:03:57.4259347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:03:57.4260134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:57.4261084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:03:57.4368537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:03:57.5271930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:03:58.8183181Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:03:58.8184106Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:03:58.8185255Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:03:58.8186065Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:03:59.1094263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:59.1094987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:03:59.4471023Z ok (4.683s) 2022-03-17T21:03:59.4471466Z 2022-03-17T21:03:59.4472226Z ---------------------------------------------------------------------- 2022-03-17T21:03:59.4472626Z Ran 1 test in 4.683s 2022-03-17T21:03:59.4472793Z 2022-03-17T21:03:59.4472884Z OK 2022-03-17T21:03:59.4473021Z 2022-03-17T21:03:59.4473154Z Generating XML reports... 2022-03-17T21:03:59.4516213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210354.xml 2022-03-17T21:04:00.7562229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:04:00.7576101Z 2022-03-17T21:04:00.7576328Z Running tests... 2022-03-17T21:04:00.7577208Z ---------------------------------------------------------------------- 2022-03-17T21:04:02.3095122Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:04:02.3542581Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39311 2022-03-17T21:04:02.3673206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39312 2022-03-17T21:04:03.4237187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:03.4238079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:03.4238864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:03.4239547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:03.4345544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:03.5252060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:05.0993511Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:05.0994338Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:05.1101705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:05.1102801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:05.4753935Z ok (4.717s) 2022-03-17T21:04:05.4754352Z 2022-03-17T21:04:05.4754984Z ---------------------------------------------------------------------- 2022-03-17T21:04:05.4755456Z Ran 1 test in 4.718s 2022-03-17T21:04:05.4755627Z 2022-03-17T21:04:05.4755720Z OK 2022-03-17T21:04:05.4755836Z 2022-03-17T21:04:05.4755974Z Generating XML reports... 2022-03-17T21:04:05.4798407Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210400.xml 2022-03-17T21:04:06.7744170Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:04:06.7759184Z 2022-03-17T21:04:06.7759459Z Running tests... 2022-03-17T21:04:06.7759953Z ---------------------------------------------------------------------- 2022-03-17T21:04:08.3280865Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:04:08.3710952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39432 2022-03-17T21:04:08.3838170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39433 2022-03-17T21:04:09.4534739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:09.4535216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:09.4536035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:09.4536735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:09.4644403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:09.5549828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:11.2079676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:11.2080164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:12.0927106Z ok (5.316s) 2022-03-17T21:04:12.0927365Z 2022-03-17T21:04:12.0927866Z ---------------------------------------------------------------------- 2022-03-17T21:04:12.0928267Z Ran 1 test in 5.317s 2022-03-17T21:04:12.0928428Z 2022-03-17T21:04:12.0928510Z OK 2022-03-17T21:04:12.0929628Z 2022-03-17T21:04:12.0929990Z Generating XML reports... 2022-03-17T21:04:12.0970127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210406.xml 2022-03-17T21:04:13.3927265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:04:13.3942045Z 2022-03-17T21:04:13.3942344Z Running tests... 2022-03-17T21:04:13.3942791Z ---------------------------------------------------------------------- 2022-03-17T21:04:14.9343948Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:04:14.9774370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39557 2022-03-17T21:04:14.9902606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39558 2022-03-17T21:04:16.1126077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:16.1126671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:16.1127436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:16.1128369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:16.1236738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:16.2140844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:17.7974909Z ok (4.403s) 2022-03-17T21:04:17.7975101Z 2022-03-17T21:04:17.7975497Z ---------------------------------------------------------------------- 2022-03-17T21:04:17.7975809Z Ran 1 test in 4.403s 2022-03-17T21:04:17.7975973Z 2022-03-17T21:04:17.7976064Z OK 2022-03-17T21:04:17.7976179Z 2022-03-17T21:04:17.7976599Z Generating XML reports... 2022-03-17T21:04:17.8018221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210413.xml 2022-03-17T21:04:19.0829233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:04:19.0842614Z 2022-03-17T21:04:19.0842897Z Running tests... 2022-03-17T21:04:19.0843365Z ---------------------------------------------------------------------- 2022-03-17T21:04:20.6198721Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:04:20.6635637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39674 2022-03-17T21:04:20.6767875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39675 2022-03-17T21:04:21.7577051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:21.7577575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:21.7578381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:21.7579038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:21.7585683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:21.7586094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:23.3838840Z ok (4.299s) 2022-03-17T21:04:23.3839065Z 2022-03-17T21:04:23.3839552Z ---------------------------------------------------------------------- 2022-03-17T21:04:23.3839866Z Ran 1 test in 4.299s 2022-03-17T21:04:23.3840027Z 2022-03-17T21:04:23.3840041Z OK 2022-03-17T21:04:23.3840178Z 2022-03-17T21:04:23.3840280Z Generating XML reports... 2022-03-17T21:04:23.3882857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210419.xml 2022-03-17T21:04:24.6846677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:04:24.6861004Z 2022-03-17T21:04:24.6861305Z Running tests... 2022-03-17T21:04:24.6861731Z ---------------------------------------------------------------------- 2022-03-17T21:04:26.2275077Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:04:26.2704730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39791 2022-03-17T21:04:26.2834293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39792 2022-03-17T21:04:27.4063646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:27.4064169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:27.4064973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:27.4065649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:27.4072429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:27.4072898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:29.0287606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:29.0288181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:29.3914268Z ok (4.705s) 2022-03-17T21:04:29.3914487Z 2022-03-17T21:04:29.3914934Z ---------------------------------------------------------------------- 2022-03-17T21:04:29.3915274Z Ran 1 test in 4.705s 2022-03-17T21:04:29.3915629Z 2022-03-17T21:04:29.3915674Z OK 2022-03-17T21:04:29.3915815Z 2022-03-17T21:04:29.3915955Z Generating XML reports... 2022-03-17T21:04:29.3958548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210424.xml 2022-03-17T21:04:30.6877777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:04:30.6892140Z 2022-03-17T21:04:30.6892628Z Running tests... 2022-03-17T21:04:30.6893011Z ---------------------------------------------------------------------- 2022-03-17T21:04:32.2230011Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:04:32.2657458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39912 2022-03-17T21:04:32.2786330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39913 2022-03-17T21:04:33.4065818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:04:33.4066382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:04:33.4067182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:33.4067886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:04:33.4175251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:04:33.5079806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:04:34.7110897Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:04:34.7954019Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:04:34.7957116Z Running test: batch_norm_net sync interval 2022-03-17T21:04:34.7957720Z 1 with iteration mapping 2022-03-17T21:04:34.7957961Z {0: 0, 1: 2} 2022-03-17T21:04:35.4326405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:35.4373215Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:35.4374087Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:35.4375304Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:35.4376507Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:35.4471398Z Running test: batch_norm_net sync interval 2022-03-17T21:04:35.4471735Z 1 with iteration mapping 2022-03-17T21:04:35.4472014Z {0: 0, 1: 3} 2022-03-17T21:04:35.4531030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:35.4600212Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:35.4601186Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:35.4602350Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:35.4603247Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:35.4683789Z Running test: batch_norm_net sync interval 2022-03-17T21:04:35.4684107Z 1 with iteration mapping 2022-03-17T21:04:35.4684383Z {0: 0, 1: 10} 2022-03-17T21:04:35.4743945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:35.5000542Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:35.5001290Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:35.5002337Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:35.5003092Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:35.5081599Z Running test: batch_norm_net sync interval 2022-03-17T21:04:35.5081943Z 1 with iteration mapping 2022-03-17T21:04:35.5082192Z {0: 5, 1: 7} 2022-03-17T21:04:36.1477764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:36.1478321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:36.1653067Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:36.1653842Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:36.1654845Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:36.1655947Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:36.1737859Z Running test: batch_norm_net sync interval 2022-03-17T21:04:36.1738204Z 1 with iteration mapping 2022-03-17T21:04:36.1738458Z {0: 5, 1: 8} 2022-03-17T21:04:36.1797269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:36.1797743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:36.1997479Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:36.1998256Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:36.1999288Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:36.2000028Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:36.2077814Z Running test: batch_norm_net sync interval 2022-03-17T21:04:36.2078157Z 1 with iteration mapping 2022-03-17T21:04:36.2078418Z {0: 5, 1: 15} 2022-03-17T21:04:36.2134924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:36.2135390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:36.2521884Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:36.2522681Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:36.2523643Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:36.2524421Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:36.2602661Z Running test: large_conv_model sync interval 2022-03-17T21:04:36.2603005Z 1 with iteration mapping 2022-03-17T21:04:36.2603258Z {0: 0, 1: 2} 2022-03-17T21:04:38.0648424Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:39.1464749Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:39.1465576Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:39.1466560Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:39.1467539Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:39.1612331Z Running test: large_conv_model sync interval 2022-03-17T21:04:39.1612659Z 1 with iteration mapping 2022-03-17T21:04:39.1612927Z {0: 0, 1: 3} 2022-03-17T21:04:40.2364966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:42.3863883Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:42.3864681Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:42.3865699Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:42.3866456Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:42.3991235Z Running test: large_conv_model sync interval 2022-03-17T21:04:42.3991570Z 1 with iteration mapping 2022-03-17T21:04:42.3991817Z {0: 0, 1: 10} 2022-03-17T21:04:43.4804099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:53.1739054Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:53.1739888Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:53.1741233Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:04:53.1742065Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:04:53.1852103Z Running test: large_conv_model sync interval 2022-03-17T21:04:53.1852486Z 1 with iteration mapping 2022-03-17T21:04:53.1852707Z {0: 5, 1: 7} 2022-03-17T21:04:54.9728003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:04:54.9728537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:01.5096879Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:01.5097689Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:01.5098761Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:01.5099791Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:01.5208865Z Running test: large_conv_model sync interval 2022-03-17T21:05:01.5209193Z 1 with iteration mapping 2022-03-17T21:05:01.5209461Z {0: 5, 1: 8} 2022-03-17T21:05:02.6137836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:02.6138377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:10.2009746Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:10.2010576Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:10.2011645Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:10.2012407Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:10.2124690Z Running test: large_conv_model sync interval 2022-03-17T21:05:10.2124996Z 1 with iteration mapping 2022-03-17T21:05:10.2125269Z {0: 5, 1: 15} 2022-03-17T21:05:11.6232239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:11.6233194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.6870730Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:26.6871703Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:26.6872765Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:26.6873506Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:26.6986758Z Running test: small_model sync interval 2022-03-17T21:05:26.6987245Z 1 with iteration mapping 2022-03-17T21:05:26.6987627Z {0: 0, 1: 2} 2022-03-17T21:05:26.7106759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.7350562Z Running test: small_model sync interval 2022-03-17T21:05:26.7351030Z 1 with iteration mapping 2022-03-17T21:05:26.7351294Z {0: 0, 1: 3} 2022-03-17T21:05:26.7439635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.7651646Z Running test: small_model sync interval 2022-03-17T21:05:26.7652232Z 1 with iteration mapping 2022-03-17T21:05:26.7652522Z {0: 0, 1: 10} 2022-03-17T21:05:26.7739686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.8260338Z Running test: small_model sync interval 2022-03-17T21:05:26.8261086Z 1 with iteration mapping 2022-03-17T21:05:26.8261707Z {0: 5, 1: 7} 2022-03-17T21:05:26.8348169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.8348633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.8718569Z Running test: small_model sync interval 2022-03-17T21:05:26.8719014Z 1 with iteration mapping 2022-03-17T21:05:26.8719268Z {0: 5, 1: 8} 2022-03-17T21:05:26.8804205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.8805031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.9218193Z Running test: small_model sync interval 2022-03-17T21:05:26.9218528Z 1 with iteration mapping 2022-03-17T21:05:26.9218788Z {0: 5, 1: 15} 2022-03-17T21:05:26.9303244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.9303738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:26.9996102Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:05:26.9996616Z 1 with iteration mapping 2022-03-17T21:05:26.9996898Z {0: 0, 1: 2} 2022-03-17T21:05:27.0073161Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0073884Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0074916Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0075687Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0099164Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.0099512Z 1 with iteration mapping 2022-03-17T21:05:27.0099775Z {0: 0, 1: 3} 2022-03-17T21:05:27.0188665Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0189477Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0190507Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0191259Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0214661Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.0215264Z 1 with iteration mapping 2022-03-17T21:05:27.0215531Z {0: 0, 1: 10} 2022-03-17T21:05:27.0419242Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0420586Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0421632Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0422384Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0445581Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.0445947Z 1 with iteration mapping 2022-03-17T21:05:27.0446206Z {0: 5, 1: 7} 2022-03-17T21:05:27.0472381Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:05:27.0598230Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0599007Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0600020Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0600792Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0624829Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.0625172Z 1 with iteration mapping 2022-03-17T21:05:27.0625420Z {0: 5, 1: 8} 2022-03-17T21:05:27.0790015Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0790799Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0791791Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.0792578Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.0816602Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.0816991Z 1 with iteration mapping 2022-03-17T21:05:27.0817232Z {0: 5, 1: 15} 2022-03-17T21:05:27.1097060Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1097932Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1098959Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1099788Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1123241Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.1123769Z 1 with iteration mapping 2022-03-17T21:05:27.1124112Z {0: 0, 1: 2} 2022-03-17T21:05:27.1150136Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:05:27.1196812Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1197611Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1198636Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1199386Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1222580Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.1222947Z 1 with iteration mapping 2022-03-17T21:05:27.1223195Z {0: 0, 1: 3} 2022-03-17T21:05:27.1313766Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1314531Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1315532Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1316289Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1339880Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.1340635Z 1 with iteration mapping 2022-03-17T21:05:27.1341038Z {0: 0, 1: 10} 2022-03-17T21:05:27.1546604Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1547532Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1548566Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1549400Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1573263Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.1573642Z 1 with iteration mapping 2022-03-17T21:05:27.1573895Z {0: 5, 1: 7} 2022-03-17T21:05:27.1721835Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1722578Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1723596Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1724337Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1749188Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.1749546Z 1 with iteration mapping 2022-03-17T21:05:27.1749795Z {0: 5, 1: 8} 2022-03-17T21:05:27.1914822Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1915571Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1916591Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.1917338Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.1941487Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:05:27.1942088Z 1 with iteration mapping 2022-03-17T21:05:27.1942372Z {0: 5, 1: 15} 2022-03-17T21:05:27.2221077Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.2221846Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.2222871Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.2223782Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.2246953Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:05:27.2247511Z 1 with iteration mapping 2022-03-17T21:05:27.2247839Z {0: 0, 1: 2} 2022-03-17T21:05:27.2274430Z registered hook 2022-03-17T21:05:27.2275733Z registered hook 2022-03-17T21:05:27.2334347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.2504185Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:05:27.2504796Z 1 with iteration mapping 2022-03-17T21:05:27.2505076Z {0: 0, 1: 3} 2022-03-17T21:05:27.2531026Z registered hook 2022-03-17T21:05:27.2531787Z registered hook 2022-03-17T21:05:27.2592410Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.2797275Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:05:27.2797999Z 1 with iteration mapping 2022-03-17T21:05:27.2798359Z {0: 0, 1: 10} 2022-03-17T21:05:27.2825536Z registered hook 2022-03-17T21:05:27.2826447Z registered hook 2022-03-17T21:05:27.2891658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.3399667Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:05:27.3400489Z 1 with iteration mapping 2022-03-17T21:05:27.3400841Z {0: 5, 1: 7} 2022-03-17T21:05:27.3426622Z registered hook 2022-03-17T21:05:27.3427514Z registered hook 2022-03-17T21:05:27.3486602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.3487631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.3870537Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:05:27.3871266Z 1 with iteration mapping 2022-03-17T21:05:27.3871846Z {0: 5, 1: 8} 2022-03-17T21:05:27.3899232Z registered hook 2022-03-17T21:05:27.3900042Z registered hook 2022-03-17T21:05:27.3961926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.3962928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.4401678Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:05:27.4402400Z 1 with iteration mapping 2022-03-17T21:05:27.4402855Z {0: 5, 1: 15} 2022-03-17T21:05:27.4431331Z registered hook 2022-03-17T21:05:27.4432013Z registered hook 2022-03-17T21:05:27.4493152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.4494123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.5218543Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:05:27.5219221Z 1 with iteration mapping 2022-03-17T21:05:27.5219695Z {0: 0, 1: 2} 2022-03-17T21:05:27.5248295Z registered hook 2022-03-17T21:05:27.5249359Z registered hook 2022-03-17T21:05:27.5265324Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T21:05:27.5265916Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T21:05:27.5311313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.5316382Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T21:05:27.5318700Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T21:05:27.5487509Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:05:27.5488253Z 1 with iteration mapping 2022-03-17T21:05:27.5488719Z {0: 0, 1: 3} 2022-03-17T21:05:27.5516098Z registered hook 2022-03-17T21:05:27.5516809Z registered hook 2022-03-17T21:05:27.5564691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.5761218Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:05:27.5761869Z 1 with iteration mapping 2022-03-17T21:05:27.5762335Z {0: 0, 1: 10} 2022-03-17T21:05:27.5789685Z registered hook 2022-03-17T21:05:27.5790404Z registered hook 2022-03-17T21:05:27.5837292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.6240377Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:05:27.6240763Z 1 with iteration mapping 2022-03-17T21:05:27.6241035Z {0: 5, 1: 7} 2022-03-17T21:05:27.6267897Z registered hook 2022-03-17T21:05:27.6268272Z registered hook 2022-03-17T21:05:27.6316001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.6316505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.6625754Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:05:27.6626149Z 1 with iteration mapping 2022-03-17T21:05:27.6626419Z {0: 5, 1: 8} 2022-03-17T21:05:27.6653288Z registered hook 2022-03-17T21:05:27.6653681Z registered hook 2022-03-17T21:05:27.6698005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.6698560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.7037887Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:05:27.7038362Z 1 with iteration mapping 2022-03-17T21:05:27.7038641Z {0: 5, 1: 15} 2022-03-17T21:05:27.7066034Z registered hook 2022-03-17T21:05:27.7066454Z registered hook 2022-03-17T21:05:27.7110630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.7111123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.7643156Z Running test: batch_norm_net sync interval 2022-03-17T21:05:27.7643526Z 2 with iteration mapping 2022-03-17T21:05:27.7643794Z {0: 0, 1: 2} 2022-03-17T21:05:27.7709490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.7747994Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.7748802Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.7749981Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.7750759Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.7834480Z Running test: batch_norm_net sync interval 2022-03-17T21:05:27.7834802Z 2 with iteration mapping 2022-03-17T21:05:27.7835060Z {0: 0, 1: 3} 2022-03-17T21:05:27.7896758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.7953294Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.7954084Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.7955083Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.7955848Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.8039528Z Running test: batch_norm_net sync interval 2022-03-17T21:05:27.8039823Z 2 with iteration mapping 2022-03-17T21:05:27.8040102Z {0: 0, 1: 10} 2022-03-17T21:05:27.8101585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.8302185Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.8302966Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.8303984Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.8304725Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.8386143Z Running test: batch_norm_net sync interval 2022-03-17T21:05:27.8386453Z 2 with iteration mapping 2022-03-17T21:05:27.8386731Z {0: 5, 1: 7} 2022-03-17T21:05:27.8447510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.8447980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.8591003Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.8591784Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.8592923Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.8593699Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.8677434Z Running test: batch_norm_net sync interval 2022-03-17T21:05:27.8677970Z 2 with iteration mapping 2022-03-17T21:05:27.8678242Z {0: 5, 1: 8} 2022-03-17T21:05:27.8737118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.8737609Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.8901295Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.8902108Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.8903122Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.8903898Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.8986535Z Running test: batch_norm_net sync interval 2022-03-17T21:05:27.8986861Z 2 with iteration mapping 2022-03-17T21:05:27.8987129Z {0: 5, 1: 15} 2022-03-17T21:05:27.9046519Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.9047015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:27.9353978Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.9354770Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.9355770Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:27.9356537Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:27.9439935Z Running test: large_conv_model sync interval 2022-03-17T21:05:27.9440407Z 3 with iteration mapping 2022-03-17T21:05:27.9440729Z {0: 0, 1: 2} 2022-03-17T21:05:29.0208882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:30.1001061Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:30.1001834Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:30.1003050Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:30.1003831Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:30.1117059Z Running test: large_conv_model sync interval 2022-03-17T21:05:30.1117410Z 3 with iteration mapping 2022-03-17T21:05:30.1117637Z {0: 0, 1: 3} 2022-03-17T21:05:31.1873420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:33.3361196Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:33.3362024Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:33.3363052Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:33.3363805Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:33.3470027Z Running test: large_conv_model sync interval 2022-03-17T21:05:33.3470827Z 3 with iteration mapping 2022-03-17T21:05:33.3471362Z {0: 0, 1: 10} 2022-03-17T21:05:34.4410932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:44.1142566Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:44.1143421Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:44.1144468Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:44.1145293Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:44.1251935Z Running test: large_conv_model sync interval 2022-03-17T21:05:44.1252291Z 3 with iteration mapping 2022-03-17T21:05:44.1252531Z {0: 5, 1: 7} 2022-03-17T21:05:45.2352183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:45.2352595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:51.7390276Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:51.7391393Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:51.7392489Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:05:51.7393305Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:05:51.7486737Z Running test: large_conv_model sync interval 2022-03-17T21:05:51.7487079Z 3 with iteration mapping 2022-03-17T21:05:51.7487316Z {0: 5, 1: 8} 2022-03-17T21:05:52.8531674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:05:52.8532367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:00.3996261Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:00.3997107Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:00.3998099Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:00.3998876Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:00.4115141Z Running test: large_conv_model sync interval 2022-03-17T21:06:00.4115478Z 3 with iteration mapping 2022-03-17T21:06:00.4115749Z {0: 5, 1: 15} 2022-03-17T21:06:01.8622352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:01.8623150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:16.9355734Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:16.9356559Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:16.9357587Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:16.9358377Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:16.9469317Z Running test: small_model sync interval 2022-03-17T21:06:16.9469708Z 4 with iteration mapping 2022-03-17T21:06:16.9469970Z {0: 0, 1: 2} 2022-03-17T21:06:16.9561242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:16.9699508Z Running test: small_model sync interval 2022-03-17T21:06:16.9699899Z 4 with iteration mapping 2022-03-17T21:06:16.9700158Z {0: 0, 1: 3} 2022-03-17T21:06:16.9800501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:16.9954721Z Running test: small_model sync interval 2022-03-17T21:06:16.9955112Z 4 with iteration mapping 2022-03-17T21:06:16.9955366Z {0: 0, 1: 10} 2022-03-17T21:06:17.0053350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.0322450Z Running test: small_model sync interval 2022-03-17T21:06:17.0323094Z 4 with iteration mapping 2022-03-17T21:06:17.0323351Z {0: 5, 1: 7} 2022-03-17T21:06:17.0416212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.0416696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.0647057Z Running test: small_model sync interval 2022-03-17T21:06:17.0647396Z 4 with iteration mapping 2022-03-17T21:06:17.0647683Z {0: 5, 1: 8} 2022-03-17T21:06:17.0733242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.0733935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.0957237Z Running test: small_model sync interval 2022-03-17T21:06:17.0957600Z 4 with iteration mapping 2022-03-17T21:06:17.0957870Z {0: 5, 1: 15} 2022-03-17T21:06:17.1046587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.1047108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.1391234Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.1391677Z 5 with iteration mapping 2022-03-17T21:06:17.1391942Z {0: 0, 1: 2} 2022-03-17T21:06:17.1456804Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.1458071Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.1459149Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.1459906Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.1483376Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.1483793Z 5 with iteration mapping 2022-03-17T21:06:17.1484037Z {0: 0, 1: 3} 2022-03-17T21:06:17.1556684Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.1557874Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.1558917Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.1560063Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.1583295Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.1583622Z 5 with iteration mapping 2022-03-17T21:06:17.1583893Z {0: 0, 1: 10} 2022-03-17T21:06:17.1724058Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.1724924Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.1725977Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.1726733Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.1750831Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.1751230Z 5 with iteration mapping 2022-03-17T21:06:17.1751504Z {0: 5, 1: 7} 2022-03-17T21:06:17.1868697Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.1869469Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.1870494Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.1871250Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.1895142Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.1895506Z 5 with iteration mapping 2022-03-17T21:06:17.1895761Z {0: 5, 1: 8} 2022-03-17T21:06:17.2019745Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2020730Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2021748Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2022524Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2046411Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.2046774Z 5 with iteration mapping 2022-03-17T21:06:17.2047048Z {0: 5, 1: 15} 2022-03-17T21:06:17.2237166Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2237949Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2239065Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2239843Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2264333Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.2264734Z 6 with iteration mapping 2022-03-17T21:06:17.2264997Z {0: 0, 1: 2} 2022-03-17T21:06:17.2328395Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2329147Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2330169Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2330917Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2354656Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.2355038Z 6 with iteration mapping 2022-03-17T21:06:17.2355284Z {0: 0, 1: 3} 2022-03-17T21:06:17.2427696Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2428460Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2429463Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2430231Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2453617Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.2453972Z 6 with iteration mapping 2022-03-17T21:06:17.2454251Z {0: 0, 1: 10} 2022-03-17T21:06:17.2597427Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2598142Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2599322Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2600076Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2623268Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.2623743Z 6 with iteration mapping 2022-03-17T21:06:17.2624012Z {0: 5, 1: 7} 2022-03-17T21:06:17.2738449Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2739228Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2740592Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2742034Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2765600Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.2766000Z 6 with iteration mapping 2022-03-17T21:06:17.2766235Z {0: 5, 1: 8} 2022-03-17T21:06:17.2928212Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2928974Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2930005Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.2930799Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.2967513Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:17.2967879Z 6 with iteration mapping 2022-03-17T21:06:17.2968146Z {0: 5, 1: 15} 2022-03-17T21:06:17.3214725Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.3215448Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.3216458Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:17.3217222Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:17.3254117Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:17.3254472Z 7 with iteration mapping 2022-03-17T21:06:17.3254704Z {0: 0, 1: 2} 2022-03-17T21:06:17.3346716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.3485213Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:17.3485549Z 7 with iteration mapping 2022-03-17T21:06:17.3485807Z {0: 0, 1: 3} 2022-03-17T21:06:17.3572623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.3721705Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:17.3722037Z 7 with iteration mapping 2022-03-17T21:06:17.3722303Z {0: 0, 1: 10} 2022-03-17T21:06:17.3808822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.4051227Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:17.4051561Z 7 with iteration mapping 2022-03-17T21:06:17.4051819Z {0: 5, 1: 7} 2022-03-17T21:06:17.4138075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.4138843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.4322492Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:17.4322875Z 7 with iteration mapping 2022-03-17T21:06:17.4323307Z {0: 5, 1: 8} 2022-03-17T21:06:17.4407844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.4408322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.4634975Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:17.4635333Z 7 with iteration mapping 2022-03-17T21:06:17.4635591Z {0: 5, 1: 15} 2022-03-17T21:06:17.4720735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.4721197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.5029968Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:17.5030371Z 8 with iteration mapping 2022-03-17T21:06:17.5030858Z {0: 0, 1: 2} 2022-03-17T21:06:17.5116779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.5262010Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:17.5262545Z 8 with iteration mapping 2022-03-17T21:06:17.5262936Z {0: 0, 1: 3} 2022-03-17T21:06:17.5349784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.5501043Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:17.5501804Z 8 with iteration mapping 2022-03-17T21:06:17.5502055Z {0: 0, 1: 10} 2022-03-17T21:06:17.5588171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.5831361Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:17.5831693Z 8 with iteration mapping 2022-03-17T21:06:17.5831959Z {0: 5, 1: 7} 2022-03-17T21:06:17.5918294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.5918793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.6102569Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:17.6102899Z 8 with iteration mapping 2022-03-17T21:06:17.6103415Z {0: 5, 1: 8} 2022-03-17T21:06:17.6189398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.6189880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.6372187Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:17.6372536Z 8 with iteration mapping 2022-03-17T21:06:17.6372798Z {0: 5, 1: 15} 2022-03-17T21:06:17.6456909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.6457433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.6740714Z Running test: batch_norm_net sync interval 2022-03-17T21:06:17.6741387Z 1 with iteration mapping 2022-03-17T21:06:17.6741853Z {0: 0, 1: 2} 2022-03-17T21:06:17.6775939Z Running test: batch_norm_net sync interval 2022-03-17T21:06:17.6776224Z 1 with iteration mapping 2022-03-17T21:06:17.6776499Z {0: 0, 1: 3} 2022-03-17T21:06:17.6806822Z Running test: batch_norm_net sync interval 2022-03-17T21:06:17.6807193Z 1 with iteration mapping 2022-03-17T21:06:17.6807433Z {0: 0, 1: 10} 2022-03-17T21:06:17.6838890Z Running test: batch_norm_net sync interval 2022-03-17T21:06:17.6839254Z 1 with iteration mapping 2022-03-17T21:06:17.6839520Z {0: 5, 1: 7} 2022-03-17T21:06:17.6905089Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.6905647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.7029645Z Running test: batch_norm_net sync interval 2022-03-17T21:06:17.7030013Z 1 with iteration mapping 2022-03-17T21:06:17.7030282Z {0: 5, 1: 8} 2022-03-17T21:06:17.7093135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.7218336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.7218803Z Running test: batch_norm_net sync interval 2022-03-17T21:06:17.7219100Z 1 with iteration mapping 2022-03-17T21:06:17.7219375Z {0: 5, 1: 15} 2022-03-17T21:06:17.7282759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.7283306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:17.7407200Z Running test: large_conv_model sync interval 2022-03-17T21:06:17.7407531Z 1 with iteration mapping 2022-03-17T21:06:17.7407795Z {0: 0, 1: 2} 2022-03-17T21:06:17.7442748Z Running test: large_conv_model sync interval 2022-03-17T21:06:17.7443161Z 1 with iteration mapping 2022-03-17T21:06:17.7443450Z {0: 0, 1: 3} 2022-03-17T21:06:17.7476948Z Running test: large_conv_model sync interval 2022-03-17T21:06:17.7477295Z 1 with iteration mapping 2022-03-17T21:06:17.7477540Z {0: 0, 1: 10} 2022-03-17T21:06:17.7510879Z Running test: large_conv_model sync interval 2022-03-17T21:06:17.7511204Z 1 with iteration mapping 2022-03-17T21:06:17.7511449Z {0: 5, 1: 7} 2022-03-17T21:06:18.8361561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:18.8362131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:23.2175464Z Running test: large_conv_model sync interval 2022-03-17T21:06:23.2175823Z 1 with iteration mapping 2022-03-17T21:06:23.2176134Z {0: 5, 1: 8} 2022-03-17T21:06:24.3101957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:24.3102798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:28.6637630Z Running test: large_conv_model sync interval 2022-03-17T21:06:28.6638021Z 1 with iteration mapping 2022-03-17T21:06:28.6638303Z {0: 5, 1: 15} 2022-03-17T21:06:29.7609033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:29.7609455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.1536497Z Running test: small_model sync interval 2022-03-17T21:06:34.1537086Z 1 with iteration mapping 2022-03-17T21:06:34.1537346Z {0: 0, 1: 2} 2022-03-17T21:06:34.1578698Z Running test: small_model sync interval 2022-03-17T21:06:34.1579025Z 1 with iteration mapping 2022-03-17T21:06:34.1579314Z {0: 0, 1: 3} 2022-03-17T21:06:34.1620521Z Running test: small_model sync interval 2022-03-17T21:06:34.1620830Z 1 with iteration mapping 2022-03-17T21:06:34.1621483Z {0: 0, 1: 10} 2022-03-17T21:06:34.1662705Z Running test: small_model sync interval 2022-03-17T21:06:34.1663008Z 1 with iteration mapping 2022-03-17T21:06:34.1663277Z {0: 5, 1: 7} 2022-03-17T21:06:34.1755888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.1756427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.1935886Z Running test: small_model sync interval 2022-03-17T21:06:34.1936196Z 1 with iteration mapping 2022-03-17T21:06:34.1936465Z {0: 5, 1: 8} 2022-03-17T21:06:34.2022438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.2022904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.2200476Z Running test: small_model sync interval 2022-03-17T21:06:34.2200802Z 1 with iteration mapping 2022-03-17T21:06:34.2201069Z {0: 5, 1: 15} 2022-03-17T21:06:34.2287898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.2288577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.2464824Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.2465181Z 1 with iteration mapping 2022-03-17T21:06:34.2465467Z {0: 0, 1: 2} 2022-03-17T21:06:34.2494307Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.2494615Z 1 with iteration mapping 2022-03-17T21:06:34.2494887Z {0: 0, 1: 3} 2022-03-17T21:06:34.2524357Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.2524740Z 1 with iteration mapping 2022-03-17T21:06:34.2524983Z {0: 0, 1: 10} 2022-03-17T21:06:34.2553444Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.2553786Z 1 with iteration mapping 2022-03-17T21:06:34.2554022Z {0: 5, 1: 7} 2022-03-17T21:06:34.2685213Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.2685739Z 1 with iteration mapping 2022-03-17T21:06:34.2686020Z {0: 5, 1: 8} 2022-03-17T21:06:34.2816357Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.2816743Z 1 with iteration mapping 2022-03-17T21:06:34.2816989Z {0: 5, 1: 15} 2022-03-17T21:06:34.2945593Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.2945929Z 1 with iteration mapping 2022-03-17T21:06:34.2946205Z {0: 0, 1: 2} 2022-03-17T21:06:34.2975280Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.2975644Z 1 with iteration mapping 2022-03-17T21:06:34.2975907Z {0: 0, 1: 3} 2022-03-17T21:06:34.3004925Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.3005279Z 1 with iteration mapping 2022-03-17T21:06:34.3005546Z {0: 0, 1: 10} 2022-03-17T21:06:34.3035282Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.3035674Z 1 with iteration mapping 2022-03-17T21:06:34.3035918Z {0: 5, 1: 7} 2022-03-17T21:06:34.3165879Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.3166243Z 1 with iteration mapping 2022-03-17T21:06:34.3166484Z {0: 5, 1: 8} 2022-03-17T21:06:34.3296622Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:06:34.3296978Z 1 with iteration mapping 2022-03-17T21:06:34.3297252Z {0: 5, 1: 15} 2022-03-17T21:06:34.3426813Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:34.3427155Z 1 with iteration mapping 2022-03-17T21:06:34.3427426Z {0: 0, 1: 2} 2022-03-17T21:06:34.3467770Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:34.3468121Z 1 with iteration mapping 2022-03-17T21:06:34.3468383Z {0: 0, 1: 3} 2022-03-17T21:06:34.3508932Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:34.3509282Z 1 with iteration mapping 2022-03-17T21:06:34.3509548Z {0: 0, 1: 10} 2022-03-17T21:06:34.3550319Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:34.3550677Z 1 with iteration mapping 2022-03-17T21:06:34.3550930Z {0: 5, 1: 7} 2022-03-17T21:06:34.3646601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.3647400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.3822841Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:34.3823165Z 1 with iteration mapping 2022-03-17T21:06:34.3823419Z {0: 5, 1: 8} 2022-03-17T21:06:34.3911396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.3912082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.4088889Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:06:34.4089479Z 1 with iteration mapping 2022-03-17T21:06:34.4089751Z {0: 5, 1: 15} 2022-03-17T21:06:34.4177084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.4177796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.4354315Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:34.4354668Z 1 with iteration mapping 2022-03-17T21:06:34.4354933Z {0: 0, 1: 2} 2022-03-17T21:06:34.4396218Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:34.4396690Z 1 with iteration mapping 2022-03-17T21:06:34.4396954Z {0: 0, 1: 3} 2022-03-17T21:06:34.4437942Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:34.4438294Z 1 with iteration mapping 2022-03-17T21:06:34.4438722Z {0: 0, 1: 10} 2022-03-17T21:06:34.4478467Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:34.4478796Z 1 with iteration mapping 2022-03-17T21:06:34.4479036Z {0: 5, 1: 7} 2022-03-17T21:06:34.4566413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.4566937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.4744898Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:34.4745255Z 1 with iteration mapping 2022-03-17T21:06:34.4745656Z {0: 5, 1: 8} 2022-03-17T21:06:34.4831731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.4832223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.5014319Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:06:34.5014710Z 1 with iteration mapping 2022-03-17T21:06:34.5014969Z {0: 5, 1: 15} 2022-03-17T21:06:34.5102351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:34.5102862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:35.0642917Z ok (124.375s) 2022-03-17T21:06:35.0643130Z 2022-03-17T21:06:35.0643532Z ---------------------------------------------------------------------- 2022-03-17T21:06:35.0643858Z Ran 1 test in 124.375s 2022-03-17T21:06:35.0644025Z 2022-03-17T21:06:35.0644127Z OK 2022-03-17T21:06:35.0644256Z 2022-03-17T21:06:35.0644373Z Generating XML reports... 2022-03-17T21:06:35.0687057Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210430.xml 2022-03-17T21:06:36.3806016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:36.3821115Z 2022-03-17T21:06:36.3821613Z Running tests... 2022-03-17T21:06:36.3822104Z ---------------------------------------------------------------------- 2022-03-17T21:06:37.9109850Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:06:37.9543473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40093 2022-03-17T21:06:37.9671067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40094 2022-03-17T21:06:39.0822539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:39.0823168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:39.0823889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:39.0824648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:39.0930912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:39.1836382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:40.7475534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:40.7476086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:40.7648812Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:40.7649562Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:40.7650895Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:40.7651667Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:40.7788027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:40.7788693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:06:40.7900080Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:06:40.7901315Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:06:41.0753023Z ok (4.693s) 2022-03-17T21:06:41.0753340Z 2022-03-17T21:06:41.0754106Z ---------------------------------------------------------------------- 2022-03-17T21:06:41.0754485Z Ran 1 test in 4.693s 2022-03-17T21:06:41.0754654Z 2022-03-17T21:06:41.0754710Z OK 2022-03-17T21:06:41.0754849Z 2022-03-17T21:06:41.0754976Z Generating XML reports... 2022-03-17T21:06:41.0796675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210636.xml 2022-03-17T21:06:42.3712618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:42.3726970Z 2022-03-17T21:06:42.3727418Z Running tests... 2022-03-17T21:06:42.3727861Z ---------------------------------------------------------------------- 2022-03-17T21:06:43.9069539Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:06:43.9499949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40214 2022-03-17T21:06:43.9629603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40215 2022-03-17T21:06:45.0737614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:45.0738175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:45.0738909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:45.0739566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:45.0848120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:45.1751009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:47.0711806Z ok (4.698s) 2022-03-17T21:06:47.0712139Z 2022-03-17T21:06:47.0712584Z ---------------------------------------------------------------------- 2022-03-17T21:06:47.0712934Z Ran 1 test in 4.698s 2022-03-17T21:06:47.0713103Z 2022-03-17T21:06:47.0714178Z OK 2022-03-17T21:06:47.0714333Z 2022-03-17T21:06:47.0714564Z Generating XML reports... 2022-03-17T21:06:47.0756183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210642.xml 2022-03-17T21:06:48.3686607Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:48.3700927Z 2022-03-17T21:06:48.3701077Z Running tests... 2022-03-17T21:06:48.3701983Z ---------------------------------------------------------------------- 2022-03-17T21:06:49.9076753Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:06:49.9502855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40335 2022-03-17T21:06:49.9631152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40336 2022-03-17T21:06:51.0498773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:51.0499383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:51.0500861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:51.0501796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:51.0507923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:51.0508766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:51.0615334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:06:51.0616074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:06:51.0616774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:06:51.0617694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:06:51.2677297Z ok (2.897s) 2022-03-17T21:06:51.2677519Z 2022-03-17T21:06:51.2678117Z ---------------------------------------------------------------------- 2022-03-17T21:06:51.2678464Z Ran 1 test in 2.898s 2022-03-17T21:06:51.2678635Z 2022-03-17T21:06:51.2678722Z OK 2022-03-17T21:06:51.2678839Z 2022-03-17T21:06:51.2678977Z Generating XML reports... 2022-03-17T21:06:51.2721464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210648.xml 2022-03-17T21:06:52.5535575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:52.5550108Z 2022-03-17T21:06:52.5550513Z Running tests... 2022-03-17T21:06:52.5550984Z ---------------------------------------------------------------------- 2022-03-17T21:06:54.1048579Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:06:54.1475573Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40456 2022-03-17T21:06:54.1603670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40457 2022-03-17T21:06:55.2004072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:55.2005148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:55.2006575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:55.2007330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:55.2111821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:06:55.3017368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:55.3227513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:06:55.3228404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:06:55.3229138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:06:55.3230072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:06:55.6658229Z ok (3.110s) 2022-03-17T21:06:55.6658696Z 2022-03-17T21:06:55.6659278Z ---------------------------------------------------------------------- 2022-03-17T21:06:55.6659653Z Ran 1 test in 3.111s 2022-03-17T21:06:55.6659782Z 2022-03-17T21:06:55.6659904Z OK 2022-03-17T21:06:55.6660029Z 2022-03-17T21:06:55.6660157Z Generating XML reports... 2022-03-17T21:06:55.6704142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210652.xml 2022-03-17T21:06:56.9770029Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:06:56.9783610Z 2022-03-17T21:06:56.9783858Z Running tests... 2022-03-17T21:06:56.9784303Z ---------------------------------------------------------------------- 2022-03-17T21:06:58.5066226Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:06:58.5494400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40577 2022-03-17T21:06:58.5623108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40578 2022-03-17T21:06:59.6618364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:06:59.6618891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:06:59.6619692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:59.6620630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:06:59.6627659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:06:59.6628448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:01.3063262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:01.3063815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:01.3140518Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:07:01.3142131Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:07:01.6702713Z ok (4.691s) 2022-03-17T21:07:01.6702944Z 2022-03-17T21:07:01.6703353Z ---------------------------------------------------------------------- 2022-03-17T21:07:01.6703702Z Ran 1 test in 4.692s 2022-03-17T21:07:01.6703851Z 2022-03-17T21:07:01.6703947Z OK 2022-03-17T21:07:01.6704092Z 2022-03-17T21:07:01.6704229Z Generating XML reports... 2022-03-17T21:07:01.6748579Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210656.xml 2022-03-17T21:07:02.9728843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:02.9743559Z 2022-03-17T21:07:02.9743781Z Running tests... 2022-03-17T21:07:02.9744270Z ---------------------------------------------------------------------- 2022-03-17T21:07:04.5106479Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:04.5535781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40698 2022-03-17T21:07:04.5663021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40699 2022-03-17T21:07:05.6362521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:05.6363308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:05.6364136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:05.6364794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:05.6472358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:05.7376252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:07.2853588Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:07:07.3044524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:07.3045064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:07:07.5740552Z ok (4.599s) 2022-03-17T21:07:07.5741012Z 2022-03-17T21:07:07.5741425Z ---------------------------------------------------------------------- 2022-03-17T21:07:07.5741777Z Ran 1 test in 4.600s 2022-03-17T21:07:07.5741927Z 2022-03-17T21:07:07.5742002Z OK 2022-03-17T21:07:07.5742137Z 2022-03-17T21:07:07.5742266Z Generating XML reports... 2022-03-17T21:07:07.5784598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210702.xml 2022-03-17T21:07:08.8698538Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:08.8713896Z 2022-03-17T21:07:08.8714138Z Running tests... 2022-03-17T21:07:08.8714548Z ---------------------------------------------------------------------- 2022-03-17T21:07:10.4309917Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:10.4739173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40819 2022-03-17T21:07:10.4874159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40820 2022-03-17T21:07:11.5829929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:11.5830444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:11.5831228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:11.5831912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:11.5941015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:11.6842093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:11.8922913Z ok (3.020s) 2022-03-17T21:07:11.8923328Z 2022-03-17T21:07:11.8924004Z ---------------------------------------------------------------------- 2022-03-17T21:07:11.8924617Z Ran 1 test in 3.021s 2022-03-17T21:07:11.8924882Z 2022-03-17T21:07:11.8925054Z OK 2022-03-17T21:07:11.8925301Z 2022-03-17T21:07:11.8925533Z Generating XML reports... 2022-03-17T21:07:11.8968593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210708.xml 2022-03-17T21:07:13.1766859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:13.1781247Z 2022-03-17T21:07:13.1781850Z Running tests... 2022-03-17T21:07:13.1782298Z ---------------------------------------------------------------------- 2022-03-17T21:07:14.7182694Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:14.7610932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40934 2022-03-17T21:07:14.7739316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40935 2022-03-17T21:07:15.8960348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:15.8960877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:15.8961632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:15.8962338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:15.9069284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:15.9973063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:16.2791884Z ok (3.101s) 2022-03-17T21:07:16.2792231Z 2022-03-17T21:07:16.2792606Z ---------------------------------------------------------------------- 2022-03-17T21:07:16.2792949Z Ran 1 test in 3.101s 2022-03-17T21:07:16.2793116Z 2022-03-17T21:07:16.2793197Z OK 2022-03-17T21:07:16.2793335Z 2022-03-17T21:07:16.2793476Z Generating XML reports... 2022-03-17T21:07:16.2835981Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210713.xml 2022-03-17T21:07:17.5601009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:17.5615073Z 2022-03-17T21:07:17.5615312Z Running tests... 2022-03-17T21:07:17.5615931Z ---------------------------------------------------------------------- 2022-03-17T21:07:19.1098787Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:19.1529455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41053 2022-03-17T21:07:19.1657359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41054 2022-03-17T21:07:20.2780371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:20.2781231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:20.2781978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:20.2782678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:20.2789924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:20.2790653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:20.4703213Z ok (2.908s) 2022-03-17T21:07:20.4703384Z 2022-03-17T21:07:20.4703745Z ---------------------------------------------------------------------- 2022-03-17T21:07:20.4704458Z Ran 1 test in 2.909s 2022-03-17T21:07:20.4704594Z 2022-03-17T21:07:20.4704666Z OK 2022-03-17T21:07:20.4704798Z 2022-03-17T21:07:20.4704938Z Generating XML reports... 2022-03-17T21:07:20.4747036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210717.xml 2022-03-17T21:07:21.7453475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:21.7467741Z 2022-03-17T21:07:21.7467892Z Running tests... 2022-03-17T21:07:21.7468698Z ---------------------------------------------------------------------- 2022-03-17T21:07:21.7489053Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:07:21.7489299Z 2022-03-17T21:07:21.7489577Z ---------------------------------------------------------------------- 2022-03-17T21:07:21.7489912Z Ran 1 test in 0.002s 2022-03-17T21:07:21.7490077Z 2022-03-17T21:07:21.7490254Z OK (skipped=1) 2022-03-17T21:07:21.7490340Z 2022-03-17T21:07:21.7490469Z Generating XML reports... 2022-03-17T21:07:21.7534418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210721.xml 2022-03-17T21:07:22.8949536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:22.8963430Z 2022-03-17T21:07:22.8963595Z Running tests... 2022-03-17T21:07:22.8964348Z ---------------------------------------------------------------------- 2022-03-17T21:07:24.4358773Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:24.4791810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41203 2022-03-17T21:07:24.4922289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41204 2022-03-17T21:07:25.5604632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:25.5605217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:25.5605952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:25.5606645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:25.5715630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:25.6616046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:25.6731241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:07:25.6731740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:07:25.6732393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:25.6733077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:25.9973188Z ok (3.101s) 2022-03-17T21:07:25.9973422Z 2022-03-17T21:07:25.9973841Z ---------------------------------------------------------------------- 2022-03-17T21:07:25.9974169Z Ran 1 test in 3.101s 2022-03-17T21:07:25.9974347Z 2022-03-17T21:07:25.9974428Z OK 2022-03-17T21:07:25.9974555Z 2022-03-17T21:07:25.9974692Z Generating XML reports... 2022-03-17T21:07:26.0018626Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210722.xml 2022-03-17T21:07:27.2852334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:27.2863654Z 2022-03-17T21:07:27.2864155Z Running tests... 2022-03-17T21:07:27.2864664Z ---------------------------------------------------------------------- 2022-03-17T21:07:28.8168844Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:28.8598264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41328 2022-03-17T21:07:28.8728300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41329 2022-03-17T21:07:29.9359662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:29.9360177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:29.9361172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:29.9361898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:29.9368336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:29.9368852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:30.0772201Z skip (2.791s) 2022-03-17T21:07:30.0772429Z 2022-03-17T21:07:30.0772788Z ---------------------------------------------------------------------- 2022-03-17T21:07:30.0773125Z Ran 1 test in 2.791s 2022-03-17T21:07:30.0773273Z 2022-03-17T21:07:30.0775077Z OK (skipped=1) 2022-03-17T21:07:30.0775280Z 2022-03-17T21:07:30.0775415Z Generating XML reports... 2022-03-17T21:07:30.0816774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210727.xml 2022-03-17T21:07:31.3665982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:31.3680175Z 2022-03-17T21:07:31.3680702Z Running tests... 2022-03-17T21:07:31.3681175Z ---------------------------------------------------------------------- 2022-03-17T21:07:32.9102968Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:32.9534802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41443 2022-03-17T21:07:32.9662821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41444 2022-03-17T21:07:34.0707277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:34.0708163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:34.0708993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:34.0709706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:34.0716074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:34.0717136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:34.2711762Z ok (2.903s) 2022-03-17T21:07:34.2712184Z 2022-03-17T21:07:34.2712938Z ---------------------------------------------------------------------- 2022-03-17T21:07:34.2713596Z Ran 1 test in 2.903s 2022-03-17T21:07:34.2713787Z 2022-03-17T21:07:34.2713889Z OK 2022-03-17T21:07:34.2714026Z 2022-03-17T21:07:34.2714163Z Generating XML reports... 2022-03-17T21:07:34.2755879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210731.xml 2022-03-17T21:07:35.5566386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:35.5580370Z 2022-03-17T21:07:35.5580750Z Running tests... 2022-03-17T21:07:35.5581809Z ---------------------------------------------------------------------- 2022-03-17T21:07:37.1035514Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:37.1462822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41558 2022-03-17T21:07:37.1591975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41559 2022-03-17T21:07:38.2137201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:38.2137696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:38.2138478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:38.2139401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:38.2247303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:38.3148364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:38.3368435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:07:38.3369010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:07:38.3369695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:38.3370368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:38.3517730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:07:38.3518302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:07:38.3519004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:07:38.3519648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:07:38.3640852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T21:07:38.3642054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T21:07:38.3642826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:07:38.3744935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:07:38.5639353Z ok (3.006s) 2022-03-17T21:07:38.5639570Z 2022-03-17T21:07:38.5640233Z ---------------------------------------------------------------------- 2022-03-17T21:07:38.5640624Z Ran 1 test in 3.006s 2022-03-17T21:07:38.5640758Z 2022-03-17T21:07:38.5640834Z OK 2022-03-17T21:07:38.5640972Z 2022-03-17T21:07:38.5641112Z Generating XML reports... 2022-03-17T21:07:38.5683123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210735.xml 2022-03-17T21:07:39.8441204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:39.8456886Z 2022-03-17T21:07:39.8457295Z Running tests... 2022-03-17T21:07:39.8457794Z ---------------------------------------------------------------------- 2022-03-17T21:07:41.3901765Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:41.4330802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41697 2022-03-17T21:07:41.4459505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41698 2022-03-17T21:07:42.5443874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:42.5444442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:42.5445468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:42.5446155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:42.5453085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:42.5453691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:42.5563183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:07:42.5563866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:07:42.5564529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:42.5565217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:42.8508381Z ok (3.005s) 2022-03-17T21:07:42.8508706Z 2022-03-17T21:07:42.8509182Z ---------------------------------------------------------------------- 2022-03-17T21:07:42.8509482Z Ran 1 test in 3.005s 2022-03-17T21:07:42.8509637Z 2022-03-17T21:07:42.8509727Z OK 2022-03-17T21:07:42.8509843Z 2022-03-17T21:07:42.8509970Z Generating XML reports... 2022-03-17T21:07:42.8552624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210739.xml 2022-03-17T21:07:44.1390005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:44.1404587Z 2022-03-17T21:07:44.1404971Z Running tests... 2022-03-17T21:07:44.1405457Z ---------------------------------------------------------------------- 2022-03-17T21:07:45.6985598Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:45.7417000Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41818 2022-03-17T21:07:45.7550711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41819 2022-03-17T21:07:46.8723729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:46.8724244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:46.8735634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:46.8736500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:46.8832978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:46.9734558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:47.1599609Z ok (3.019s) 2022-03-17T21:07:47.1599974Z 2022-03-17T21:07:47.1600584Z ---------------------------------------------------------------------- 2022-03-17T21:07:47.1601202Z Ran 1 test in 3.019s 2022-03-17T21:07:47.1601502Z 2022-03-17T21:07:47.1601632Z OK 2022-03-17T21:07:47.1601871Z 2022-03-17T21:07:47.1602112Z Generating XML reports... 2022-03-17T21:07:47.1644902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210744.xml 2022-03-17T21:07:48.4464219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:48.4478674Z 2022-03-17T21:07:48.4479084Z Running tests... 2022-03-17T21:07:48.4479601Z ---------------------------------------------------------------------- 2022-03-17T21:07:49.9995354Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:50.0425702Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41933 2022-03-17T21:07:50.0554098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41934 2022-03-17T21:07:51.1635809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:51.1636329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:51.1637131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:51.1638054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:51.1745665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:51.2649014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:51.6604927Z ok (3.212s) 2022-03-17T21:07:51.6605423Z 2022-03-17T21:07:51.6605872Z ---------------------------------------------------------------------- 2022-03-17T21:07:51.6606227Z Ran 1 test in 3.213s 2022-03-17T21:07:51.6606391Z 2022-03-17T21:07:51.6606472Z OK 2022-03-17T21:07:51.6606611Z 2022-03-17T21:07:51.6606743Z Generating XML reports... 2022-03-17T21:07:51.6648977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210748.xml 2022-03-17T21:07:52.9545826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:52.9559891Z 2022-03-17T21:07:52.9560478Z Running tests... 2022-03-17T21:07:52.9560996Z ---------------------------------------------------------------------- 2022-03-17T21:07:54.4674540Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:54.5098485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42048 2022-03-17T21:07:54.5225993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42049 2022-03-17T21:07:55.5783740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:55.5784272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:55.5785068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:55.5785768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:55.5792432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:55.5792919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:55.6002394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:07:55.6002893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:07:55.6003578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:55.6004261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:55.8270590Z ok (2.871s) 2022-03-17T21:07:55.8270856Z 2022-03-17T21:07:55.8271220Z ---------------------------------------------------------------------- 2022-03-17T21:07:55.8271558Z Ran 1 test in 2.871s 2022-03-17T21:07:55.8271705Z 2022-03-17T21:07:55.8271811Z OK 2022-03-17T21:07:55.8271991Z 2022-03-17T21:07:55.8273048Z Generating XML reports... 2022-03-17T21:07:55.8314195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210752.xml 2022-03-17T21:07:57.1146855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:07:57.1161056Z 2022-03-17T21:07:57.1161317Z Running tests... 2022-03-17T21:07:57.1161737Z ---------------------------------------------------------------------- 2022-03-17T21:07:58.6635370Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:07:58.7076101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42169 2022-03-17T21:07:58.7203506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42170 2022-03-17T21:07:59.8301522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:07:59.8302088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:07:59.8302876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:59.8303580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:07:59.8310138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:07:59.8310623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:07:59.8519349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:07:59.8519838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:07:59.8520524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:07:59.8521212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:08:00.1251082Z ok (3.009s) 2022-03-17T21:08:00.1251323Z 2022-03-17T21:08:00.1251696Z ---------------------------------------------------------------------- 2022-03-17T21:08:00.1252035Z Ran 1 test in 3.009s 2022-03-17T21:08:00.1252203Z 2022-03-17T21:08:00.1252298Z OK 2022-03-17T21:08:00.1252417Z 2022-03-17T21:08:00.1252549Z Generating XML reports... 2022-03-17T21:08:00.1294902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210757.xml 2022-03-17T21:08:01.4220009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:01.4235949Z 2022-03-17T21:08:01.4236214Z Running tests... 2022-03-17T21:08:01.4236692Z ---------------------------------------------------------------------- 2022-03-17T21:08:02.9630077Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:08:03.0060650Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42290 2022-03-17T21:08:03.0190242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42291 2022-03-17T21:08:04.0755818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:04.0756362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:04.0757144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:04.0757823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:04.0865951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:04.1769924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:06.0268230Z ok (4.603s) 2022-03-17T21:08:06.0268603Z 2022-03-17T21:08:06.0268995Z ---------------------------------------------------------------------- 2022-03-17T21:08:06.0269651Z Ran 1 test in 4.603s 2022-03-17T21:08:06.0269816Z 2022-03-17T21:08:06.0269910Z OK 2022-03-17T21:08:06.0270046Z 2022-03-17T21:08:06.0270183Z Generating XML reports... 2022-03-17T21:08:06.0314599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210801.xml 2022-03-17T21:08:07.3301266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:07.3315122Z 2022-03-17T21:08:07.3315569Z Running tests... 2022-03-17T21:08:07.3316047Z ---------------------------------------------------------------------- 2022-03-17T21:08:08.8877339Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:08:08.9307974Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42411 2022-03-17T21:08:08.9439559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42412 2022-03-17T21:08:10.0261899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:10.0262445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:10.0263181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:10.0263879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:10.0369770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:10.1272300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:10.3489799Z ok (3.017s) 2022-03-17T21:08:10.3490222Z 2022-03-17T21:08:10.3490651Z ---------------------------------------------------------------------- 2022-03-17T21:08:10.3490935Z Ran 1 test in 3.017s 2022-03-17T21:08:10.3491122Z 2022-03-17T21:08:10.3491214Z OK 2022-03-17T21:08:10.3491330Z 2022-03-17T21:08:10.3491463Z Generating XML reports... 2022-03-17T21:08:10.3533666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210807.xml 2022-03-17T21:08:11.6366179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:11.6380262Z 2022-03-17T21:08:11.6380543Z Running tests... 2022-03-17T21:08:11.6381253Z ---------------------------------------------------------------------- 2022-03-17T21:08:13.2117583Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:08:13.2550973Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42526 2022-03-17T21:08:13.2677455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42527 2022-03-17T21:08:14.3502225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:14.3502781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:14.3503570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:14.3504257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:14.3610490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:14.4513664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:14.6725917Z ok (3.034s) 2022-03-17T21:08:14.6726122Z 2022-03-17T21:08:14.6726537Z ---------------------------------------------------------------------- 2022-03-17T21:08:14.6726878Z Ran 1 test in 3.034s 2022-03-17T21:08:14.6727050Z 2022-03-17T21:08:14.6727131Z OK 2022-03-17T21:08:14.6727264Z 2022-03-17T21:08:14.6727618Z Generating XML reports... 2022-03-17T21:08:14.6769396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210811.xml 2022-03-17T21:08:15.9581790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:15.9597686Z 2022-03-17T21:08:15.9597851Z Running tests... 2022-03-17T21:08:15.9598495Z ---------------------------------------------------------------------- 2022-03-17T21:08:17.5077863Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:08:17.5514739Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42641 2022-03-17T21:08:17.5647220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42642 2022-03-17T21:08:18.6787709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:18.6788276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:18.6789058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:18.6789749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:18.6898253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:18.7798427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:19.0698184Z ok (3.110s) 2022-03-17T21:08:19.0698529Z 2022-03-17T21:08:19.0698934Z ---------------------------------------------------------------------- 2022-03-17T21:08:19.0699280Z Ran 1 test in 3.110s 2022-03-17T21:08:19.0699453Z 2022-03-17T21:08:19.0699540Z OK 2022-03-17T21:08:19.0699657Z 2022-03-17T21:08:19.0699791Z Generating XML reports... 2022-03-17T21:08:19.0742896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210815.xml 2022-03-17T21:08:20.3610227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:20.3624493Z 2022-03-17T21:08:20.3624962Z Running tests... 2022-03-17T21:08:20.3625440Z ---------------------------------------------------------------------- 2022-03-17T21:08:21.8972800Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:08:21.9398121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42760 2022-03-17T21:08:21.9525722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42761 2022-03-17T21:08:23.0572439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:23.0573167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:23.0573952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:23.0574644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:23.0680232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:23.1584318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:23.4575357Z ok (3.095s) 2022-03-17T21:08:23.4580050Z 2022-03-17T21:08:23.4580684Z ---------------------------------------------------------------------- 2022-03-17T21:08:23.4581233Z Ran 1 test in 3.095s 2022-03-17T21:08:23.4581451Z 2022-03-17T21:08:23.4581546Z OK 2022-03-17T21:08:23.4581672Z 2022-03-17T21:08:23.4581797Z Generating XML reports... 2022-03-17T21:08:23.4637851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210820.xml 2022-03-17T21:08:24.7507820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:24.7522510Z 2022-03-17T21:08:24.7522664Z Running tests... 2022-03-17T21:08:24.7523525Z ---------------------------------------------------------------------- 2022-03-17T21:08:26.2959032Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:08:26.3389725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42879 2022-03-17T21:08:26.3518449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42880 2022-03-17T21:08:27.4903254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:27.4903850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:27.4904628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:27.4905336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:27.5010380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:27.5013553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:08:27.5913426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:27.5917306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:08:27.5918246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:08:27.5930407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:08:27.6128032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:08:27.6129108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:08:27.6129843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:08:27.6130504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:08:30.0705056Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:08:45.6863368Z ok (20.934s) 2022-03-17T21:08:45.6863751Z 2022-03-17T21:08:45.6864489Z ---------------------------------------------------------------------- 2022-03-17T21:08:45.6865187Z Ran 1 test in 20.934s 2022-03-17T21:08:45.6865365Z 2022-03-17T21:08:45.6865468Z OK 2022-03-17T21:08:45.6865597Z 2022-03-17T21:08:45.6865733Z Generating XML reports... 2022-03-17T21:08:45.6907905Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210824.xml 2022-03-17T21:08:46.9890417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:08:46.9905050Z 2022-03-17T21:08:46.9905361Z Running tests... 2022-03-17T21:08:46.9905798Z ---------------------------------------------------------------------- 2022-03-17T21:08:48.5329493Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:08:48.5755167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43012 2022-03-17T21:08:48.5883959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43013 2022-03-17T21:08:49.7179235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:08:49.7180001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:08:49.7181048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:49.7181743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:08:49.7188175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:08:49.7190263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:08:49.7191022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:08:49.7193761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:08:49.7194432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:08:49.7296271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:08:49.7503326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:08:49.7503780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:08:49.7504724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:08:49.7505456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:08:52.2086914Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:08:52.2087427Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:09:07.7257360Z ok (20.735s) 2022-03-17T21:09:07.7257838Z 2022-03-17T21:09:07.7258283Z ---------------------------------------------------------------------- 2022-03-17T21:09:07.7258512Z Ran 1 test in 20.735s 2022-03-17T21:09:07.7258730Z 2022-03-17T21:09:07.7259635Z OK 2022-03-17T21:09:07.7259881Z 2022-03-17T21:09:07.7260095Z Generating XML reports... 2022-03-17T21:09:07.7302661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210846.xml 2022-03-17T21:09:09.0224393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:09.0238658Z 2022-03-17T21:09:09.0239096Z Running tests... 2022-03-17T21:09:09.0239599Z ---------------------------------------------------------------------- 2022-03-17T21:09:10.5517878Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:10.5944080Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43145 2022-03-17T21:09:10.6075682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43146 2022-03-17T21:09:11.6638116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:11.6638650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:11.6639417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:11.6640106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:11.6646821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:11.6647307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:11.8120664Z skip (2.788s) 2022-03-17T21:09:11.8121191Z 2022-03-17T21:09:11.8121588Z ---------------------------------------------------------------------- 2022-03-17T21:09:11.8121872Z Ran 1 test in 2.788s 2022-03-17T21:09:11.8122043Z 2022-03-17T21:09:11.8122151Z OK (skipped=1) 2022-03-17T21:09:11.8124145Z 2022-03-17T21:09:11.8124515Z Generating XML reports... 2022-03-17T21:09:11.8167787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210909.xml 2022-03-17T21:09:13.0940880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:13.0954969Z 2022-03-17T21:09:13.0955242Z Running tests... 2022-03-17T21:09:13.0955965Z ---------------------------------------------------------------------- 2022-03-17T21:09:14.6414827Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:14.6847197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43260 2022-03-17T21:09:14.6975035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43261 2022-03-17T21:09:15.7821666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:15.7822219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:15.7823009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:15.7823704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:15.7830528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:15.7831287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:17.7988573Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-03-17T21:09:18.1058797Z ok (5.010s) 2022-03-17T21:09:18.1059038Z 2022-03-17T21:09:18.1059860Z ---------------------------------------------------------------------- 2022-03-17T21:09:18.1060507Z Ran 1 test in 5.010s 2022-03-17T21:09:18.1060678Z 2022-03-17T21:09:18.1060788Z OK 2022-03-17T21:09:18.1060907Z 2022-03-17T21:09:18.1061042Z Generating XML reports... 2022-03-17T21:09:18.1103357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210913.xml 2022-03-17T21:09:19.4118072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:19.4132411Z 2022-03-17T21:09:19.4132877Z Running tests... 2022-03-17T21:09:19.4133293Z ---------------------------------------------------------------------- 2022-03-17T21:09:20.9433392Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:20.9858155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43375 2022-03-17T21:09:20.9987349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43376 2022-03-17T21:09:22.1157997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:22.1158520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:22.1159257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:22.1159958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:22.1166658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:22.1167235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:22.1374076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:09:22.1374843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:09:22.1375537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:09:22.1376217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:09:22.1378384Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-03-17T21:09:22.1378800Z No ranks successfully processed in monitoredBarrier. 2022-03-17T21:09:22.1409016Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-03-17T21:09:22.4036423Z ok (2.990s) 2022-03-17T21:09:22.4036787Z 2022-03-17T21:09:22.4037451Z ---------------------------------------------------------------------- 2022-03-17T21:09:22.4038036Z Ran 1 test in 2.990s 2022-03-17T21:09:22.4038385Z 2022-03-17T21:09:22.4038537Z OK 2022-03-17T21:09:22.4038758Z 2022-03-17T21:09:22.4039014Z Generating XML reports... 2022-03-17T21:09:22.4082663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210919.xml 2022-03-17T21:09:23.6590515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:23.6605260Z 2022-03-17T21:09:23.6605516Z Running tests... 2022-03-17T21:09:23.6605947Z ---------------------------------------------------------------------- 2022-03-17T21:09:25.1937324Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:25.2368088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43496 2022-03-17T21:09:25.2496384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43497 2022-03-17T21:09:26.3380122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:26.3380913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:26.3381719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:26.3382392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:26.3388867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:26.3389654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:26.3498557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:09:26.3499023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:09:26.3499717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:09:26.3500612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:09:26.4609416Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:09:26.7547927Z ok (3.094s) 2022-03-17T21:09:26.7548149Z 2022-03-17T21:09:26.7548605Z ---------------------------------------------------------------------- 2022-03-17T21:09:26.7548922Z Ran 1 test in 3.094s 2022-03-17T21:09:26.7549098Z 2022-03-17T21:09:26.7549195Z OK 2022-03-17T21:09:26.7549340Z 2022-03-17T21:09:26.7549475Z Generating XML reports... 2022-03-17T21:09:26.7592357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210923.xml 2022-03-17T21:09:28.0359555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:28.0373413Z 2022-03-17T21:09:28.0373866Z Running tests... 2022-03-17T21:09:28.0374352Z ---------------------------------------------------------------------- 2022-03-17T21:09:29.5857046Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:29.6284400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43617 2022-03-17T21:09:29.6412662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43618 2022-03-17T21:09:30.7110007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:30.7110756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:30.7111538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:30.7112227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:30.7217515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:30.8124244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:30.9456483Z skip (2.908s) 2022-03-17T21:09:30.9456701Z 2022-03-17T21:09:30.9457095Z ---------------------------------------------------------------------- 2022-03-17T21:09:30.9457436Z Ran 1 test in 2.908s 2022-03-17T21:09:30.9457579Z 2022-03-17T21:09:30.9457678Z OK (skipped=1) 2022-03-17T21:09:30.9457816Z 2022-03-17T21:09:30.9457943Z Generating XML reports... 2022-03-17T21:09:30.9500489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210928.xml 2022-03-17T21:09:32.2253480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:32.2268136Z 2022-03-17T21:09:32.2268234Z Running tests... 2022-03-17T21:09:32.2269355Z ---------------------------------------------------------------------- 2022-03-17T21:09:32.2296707Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:09:32.2297117Z 2022-03-17T21:09:32.2297461Z ---------------------------------------------------------------------- 2022-03-17T21:09:32.2297748Z Ran 1 test in 0.003s 2022-03-17T21:09:32.2298044Z 2022-03-17T21:09:32.2298224Z OK (skipped=1) 2022-03-17T21:09:32.2298442Z 2022-03-17T21:09:32.2298567Z Generating XML reports... 2022-03-17T21:09:32.2342150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210932.xml 2022-03-17T21:09:33.3821837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:33.3836319Z 2022-03-17T21:09:33.3836695Z Running tests... 2022-03-17T21:09:33.3837144Z ---------------------------------------------------------------------- 2022-03-17T21:09:33.3867190Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:09:33.3867700Z 2022-03-17T21:09:33.3867957Z ---------------------------------------------------------------------- 2022-03-17T21:09:33.3868271Z Ran 1 test in 0.003s 2022-03-17T21:09:33.3868411Z 2022-03-17T21:09:33.3868520Z OK (skipped=1) 2022-03-17T21:09:33.3868678Z 2022-03-17T21:09:33.3868784Z Generating XML reports... 2022-03-17T21:09:33.3914641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210933.xml 2022-03-17T21:09:34.5328950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:34.5343516Z 2022-03-17T21:09:34.5343924Z Running tests... 2022-03-17T21:09:34.5344403Z ---------------------------------------------------------------------- 2022-03-17T21:09:34.5369552Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:09:34.5370119Z 2022-03-17T21:09:34.5370447Z ---------------------------------------------------------------------- 2022-03-17T21:09:34.5370750Z Ran 1 test in 0.003s 2022-03-17T21:09:34.5370885Z 2022-03-17T21:09:34.5370996Z OK (skipped=1) 2022-03-17T21:09:34.5371130Z 2022-03-17T21:09:34.5371239Z Generating XML reports... 2022-03-17T21:09:34.5414883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210934.xml 2022-03-17T21:09:35.6807099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:35.6821126Z 2022-03-17T21:09:35.6821394Z Running tests... 2022-03-17T21:09:35.6822055Z ---------------------------------------------------------------------- 2022-03-17T21:09:35.6854643Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:09:35.6854960Z 2022-03-17T21:09:35.6855257Z ---------------------------------------------------------------------- 2022-03-17T21:09:35.6855539Z Ran 1 test in 0.003s 2022-03-17T21:09:35.6855695Z 2022-03-17T21:09:35.6855810Z OK (skipped=1) 2022-03-17T21:09:35.6855941Z 2022-03-17T21:09:35.6856051Z Generating XML reports... 2022-03-17T21:09:35.6899997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210935.xml 2022-03-17T21:09:36.8278974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:36.8292858Z 2022-03-17T21:09:36.8292978Z Running tests... 2022-03-17T21:09:36.8293729Z ---------------------------------------------------------------------- 2022-03-17T21:09:36.8320091Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:09:36.8320365Z 2022-03-17T21:09:36.8320669Z ---------------------------------------------------------------------- 2022-03-17T21:09:36.8320943Z Ran 1 test in 0.003s 2022-03-17T21:09:36.8321161Z 2022-03-17T21:09:36.8321275Z OK (skipped=1) 2022-03-17T21:09:36.8321430Z 2022-03-17T21:09:36.8321555Z Generating XML reports... 2022-03-17T21:09:36.8366160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210936.xml 2022-03-17T21:09:37.9800054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:37.9813821Z 2022-03-17T21:09:37.9813933Z Running tests... 2022-03-17T21:09:37.9814681Z ---------------------------------------------------------------------- 2022-03-17T21:09:37.9838413Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:09:37.9838680Z 2022-03-17T21:09:37.9838997Z ---------------------------------------------------------------------- 2022-03-17T21:09:37.9839315Z Ran 1 test in 0.002s 2022-03-17T21:09:37.9839491Z 2022-03-17T21:09:37.9839606Z OK (skipped=1) 2022-03-17T21:09:37.9839763Z 2022-03-17T21:09:37.9839869Z Generating XML reports... 2022-03-17T21:09:37.9884255Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210937.xml 2022-03-17T21:09:39.1348030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:39.1362208Z 2022-03-17T21:09:39.1362627Z Running tests... 2022-03-17T21:09:39.1363184Z ---------------------------------------------------------------------- 2022-03-17T21:09:39.1390638Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:09:39.1391126Z 2022-03-17T21:09:39.1391438Z ---------------------------------------------------------------------- 2022-03-17T21:09:39.1391789Z Ran 1 test in 0.003s 2022-03-17T21:09:39.1391904Z 2022-03-17T21:09:39.1392027Z OK (skipped=1) 2022-03-17T21:09:39.1392287Z 2022-03-17T21:09:39.1392517Z Generating XML reports... 2022-03-17T21:09:39.1436916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210939.xml 2022-03-17T21:09:40.2990063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:40.3004500Z 2022-03-17T21:09:40.3004641Z Running tests... 2022-03-17T21:09:40.3005289Z ---------------------------------------------------------------------- 2022-03-17T21:09:40.3029425Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:09:40.3029728Z 2022-03-17T21:09:40.3029973Z ---------------------------------------------------------------------- 2022-03-17T21:09:40.3030304Z Ran 1 test in 0.002s 2022-03-17T21:09:40.3030438Z 2022-03-17T21:09:40.3030845Z OK (skipped=1) 2022-03-17T21:09:40.3030928Z 2022-03-17T21:09:40.3031084Z Generating XML reports... 2022-03-17T21:09:40.3075356Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210940.xml 2022-03-17T21:09:41.4538791Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:41.4553518Z 2022-03-17T21:09:41.4554074Z Running tests... 2022-03-17T21:09:41.4554607Z ---------------------------------------------------------------------- 2022-03-17T21:09:43.0192569Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:43.0619534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44012 2022-03-17T21:09:43.0749572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44013 2022-03-17T21:09:44.1691010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:44.1691556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:44.1692367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:44.1693055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:44.1700055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:44.1701023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:44.3795901Z ok (2.924s) 2022-03-17T21:09:44.3796367Z 2022-03-17T21:09:44.3796788Z ---------------------------------------------------------------------- 2022-03-17T21:09:44.3797114Z Ran 1 test in 2.924s 2022-03-17T21:09:44.3797279Z 2022-03-17T21:09:44.3797352Z OK 2022-03-17T21:09:44.3797505Z 2022-03-17T21:09:44.3797639Z Generating XML reports... 2022-03-17T21:09:44.3839255Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210941.xml 2022-03-17T21:09:45.6686396Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:45.6700750Z 2022-03-17T21:09:45.6701163Z Running tests... 2022-03-17T21:09:45.6701592Z ---------------------------------------------------------------------- 2022-03-17T21:09:47.1978151Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:47.2407576Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44127 2022-03-17T21:09:47.2536486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44128 2022-03-17T21:09:48.3490232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:48.3490773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:48.3491559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:48.3492492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:48.3598342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:48.4501154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:48.6589498Z ok (2.989s) 2022-03-17T21:09:48.6589718Z 2022-03-17T21:09:48.6590119Z ---------------------------------------------------------------------- 2022-03-17T21:09:48.6590445Z Ran 1 test in 2.989s 2022-03-17T21:09:48.6590604Z 2022-03-17T21:09:48.6590752Z OK 2022-03-17T21:09:48.6590886Z 2022-03-17T21:09:48.6591273Z Generating XML reports... 2022-03-17T21:09:48.6633603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210945.xml 2022-03-17T21:09:49.9424387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:49.9438384Z 2022-03-17T21:09:49.9438652Z Running tests... 2022-03-17T21:09:49.9439099Z ---------------------------------------------------------------------- 2022-03-17T21:09:49.9460788Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:09:49.9461056Z 2022-03-17T21:09:49.9461318Z ---------------------------------------------------------------------- 2022-03-17T21:09:49.9461677Z Ran 1 test in 0.002s 2022-03-17T21:09:49.9461846Z 2022-03-17T21:09:49.9461929Z OK (skipped=1) 2022-03-17T21:09:49.9462095Z 2022-03-17T21:09:49.9462206Z Generating XML reports... 2022-03-17T21:09:49.9507143Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210949.xml 2022-03-17T21:09:51.0912229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:51.0926178Z 2022-03-17T21:09:51.0926328Z Running tests... 2022-03-17T21:09:51.0927353Z ---------------------------------------------------------------------- 2022-03-17T21:09:51.0948188Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:09:51.0948571Z 2022-03-17T21:09:51.0948845Z ---------------------------------------------------------------------- 2022-03-17T21:09:51.0949118Z Ran 1 test in 0.002s 2022-03-17T21:09:51.0949410Z 2022-03-17T21:09:51.0949599Z OK (skipped=1) 2022-03-17T21:09:51.0949785Z 2022-03-17T21:09:51.0949892Z Generating XML reports... 2022-03-17T21:09:51.0993521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210951.xml 2022-03-17T21:09:52.2416145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:52.2430820Z 2022-03-17T21:09:52.2431320Z Running tests... 2022-03-17T21:09:52.2431794Z ---------------------------------------------------------------------- 2022-03-17T21:09:53.7793204Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:53.8214549Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44312 2022-03-17T21:09:53.8340757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44313 2022-03-17T21:09:54.8946052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:09:54.8946574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:09:54.8947330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:54.8948019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:09:54.8954765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:09:54.8955626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:09:56.8417077Z ok (4.598s) 2022-03-17T21:09:56.8417358Z 2022-03-17T21:09:56.8417891Z ---------------------------------------------------------------------- 2022-03-17T21:09:56.8418216Z Ran 1 test in 4.599s 2022-03-17T21:09:56.8418388Z 2022-03-17T21:09:56.8418481Z OK 2022-03-17T21:09:56.8418597Z 2022-03-17T21:09:56.8418731Z Generating XML reports... 2022-03-17T21:09:56.8460233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210952.xml 2022-03-17T21:09:58.1406074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:09:58.1421856Z 2022-03-17T21:09:58.1422247Z Running tests... 2022-03-17T21:09:58.1422752Z ---------------------------------------------------------------------- 2022-03-17T21:09:59.6638873Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:09:59.7067421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44433 2022-03-17T21:09:59.7195656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44434 2022-03-17T21:10:00.8232887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:00.8233859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:00.8235281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:00.8236071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:00.8342316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:00.9248200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:02.8275920Z ok (4.685s) 2022-03-17T21:10:02.8276194Z 2022-03-17T21:10:02.8276594Z ---------------------------------------------------------------------- 2022-03-17T21:10:02.8276929Z Ran 1 test in 4.685s 2022-03-17T21:10:02.8277087Z 2022-03-17T21:10:02.8277181Z OK 2022-03-17T21:10:02.8277276Z 2022-03-17T21:10:02.8319049Z Generating XML reports... 2022-03-17T21:10:02.8319675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210958.xml 2022-03-17T21:10:04.1331152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:04.1345983Z 2022-03-17T21:10:04.1346155Z Running tests... 2022-03-17T21:10:04.1346630Z ---------------------------------------------------------------------- 2022-03-17T21:10:05.6745082Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:05.7175690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44554 2022-03-17T21:10:05.7304300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44555 2022-03-17T21:10:06.8453313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:06.8453862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:06.8454635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:06.8455375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:06.8462403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:06.8463257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:09.5398352Z ok (5.405s) 2022-03-17T21:10:09.5398585Z 2022-03-17T21:10:09.5398955Z ---------------------------------------------------------------------- 2022-03-17T21:10:09.5399310Z Ran 1 test in 5.405s 2022-03-17T21:10:09.5399463Z 2022-03-17T21:10:09.5401000Z OK 2022-03-17T21:10:09.5401195Z 2022-03-17T21:10:09.5402309Z Generating XML reports... 2022-03-17T21:10:09.5442110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211004.xml 2022-03-17T21:10:10.8373888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:10.8388532Z 2022-03-17T21:10:10.8388811Z Running tests... 2022-03-17T21:10:10.8389827Z ---------------------------------------------------------------------- 2022-03-17T21:10:12.3738988Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:12.4167320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44672 2022-03-17T21:10:12.4296962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44673 2022-03-17T21:10:13.5161386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:13.5161933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:13.5162706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:13.5163397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:13.5272172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:13.6176151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:15.1916592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:10:15.1917127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:10:15.1939799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:10:15.1940281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:10:15.6377208Z ok (4.799s) 2022-03-17T21:10:15.6377462Z 2022-03-17T21:10:15.6377841Z ---------------------------------------------------------------------- 2022-03-17T21:10:15.6378177Z Ran 1 test in 4.799s 2022-03-17T21:10:15.6378325Z 2022-03-17T21:10:15.6378421Z OK 2022-03-17T21:10:15.6378554Z 2022-03-17T21:10:15.6378700Z Generating XML reports... 2022-03-17T21:10:15.6420209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211010.xml 2022-03-17T21:10:16.9359110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:16.9372903Z 2022-03-17T21:10:16.9373296Z Running tests... 2022-03-17T21:10:16.9373795Z ---------------------------------------------------------------------- 2022-03-17T21:10:18.4705938Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:18.5136840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44793 2022-03-17T21:10:18.5265964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44794 2022-03-17T21:10:19.6407313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:19.6408380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:19.6409781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:19.6411143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:19.6414850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:19.6417654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:19.6622566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:10:19.6623538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:10:19.6624875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:19.6626578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:19.8312255Z ok (2.894s) 2022-03-17T21:10:19.8312471Z 2022-03-17T21:10:19.8312901Z ---------------------------------------------------------------------- 2022-03-17T21:10:19.8313218Z Ran 1 test in 2.894s 2022-03-17T21:10:19.8313385Z 2022-03-17T21:10:19.8313487Z OK 2022-03-17T21:10:19.8313615Z 2022-03-17T21:10:19.8313750Z Generating XML reports... 2022-03-17T21:10:19.8355464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211016.xml 2022-03-17T21:10:21.1210206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:21.1224873Z 2022-03-17T21:10:21.1225184Z Running tests... 2022-03-17T21:10:21.1225604Z ---------------------------------------------------------------------- 2022-03-17T21:10:22.6651666Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:22.7081779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44918 2022-03-17T21:10:22.7210347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44919 2022-03-17T21:10:23.7767412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:23.7768024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:23.7768837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:23.7769478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:23.7777630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:23.7784664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:23.7794242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:10:23.7885774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:10:23.7886491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:23.7896730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:24.0255878Z ok (2.903s) 2022-03-17T21:10:24.0256213Z 2022-03-17T21:10:24.0256797Z ---------------------------------------------------------------------- 2022-03-17T21:10:24.0257148Z Ran 1 test in 2.903s 2022-03-17T21:10:24.0257234Z 2022-03-17T21:10:24.0257347Z OK 2022-03-17T21:10:24.0257450Z 2022-03-17T21:10:24.0257584Z Generating XML reports... 2022-03-17T21:10:24.0299127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211021.xml 2022-03-17T21:10:25.3235798Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:25.3249584Z 2022-03-17T21:10:25.3249789Z Running tests... 2022-03-17T21:10:25.3250231Z ---------------------------------------------------------------------- 2022-03-17T21:10:26.8799563Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:26.9229445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45043 2022-03-17T21:10:26.9360158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45044 2022-03-17T21:10:28.0236533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:28.0237269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:28.0238063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:28.0238751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:28.0244473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:28.0244961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:28.0352441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:10:28.0352955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:10:28.0353600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:28.0354293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:28.2406458Z ok (2.915s) 2022-03-17T21:10:28.2406697Z 2022-03-17T21:10:28.2407067Z ---------------------------------------------------------------------- 2022-03-17T21:10:28.2407419Z Ran 1 test in 2.916s 2022-03-17T21:10:28.2407582Z 2022-03-17T21:10:28.2407678Z OK 2022-03-17T21:10:28.2407786Z 2022-03-17T21:10:28.2407920Z Generating XML reports... 2022-03-17T21:10:28.2451141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211025.xml 2022-03-17T21:10:29.5248859Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:29.5263185Z 2022-03-17T21:10:29.5263393Z Running tests... 2022-03-17T21:10:29.5263840Z ---------------------------------------------------------------------- 2022-03-17T21:10:31.0590969Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:31.1020675Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45168 2022-03-17T21:10:31.1148460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45169 2022-03-17T21:10:32.2179943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:32.2180925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:32.2181706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:32.2182380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:32.2290281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:32.3191163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:32.3399636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:10:32.3400660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:10:32.3402460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:32.3403786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:10:32.5196940Z ok (2.993s) 2022-03-17T21:10:32.5197339Z 2022-03-17T21:10:32.5198005Z ---------------------------------------------------------------------- 2022-03-17T21:10:32.5198477Z Ran 1 test in 2.993s 2022-03-17T21:10:32.5198717Z 2022-03-17T21:10:32.5198818Z OK 2022-03-17T21:10:32.5198954Z 2022-03-17T21:10:32.5199109Z Generating XML reports... 2022-03-17T21:10:32.5240700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211029.xml 2022-03-17T21:10:33.8047443Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:33.8061304Z 2022-03-17T21:10:33.8061740Z Running tests... 2022-03-17T21:10:33.8062218Z ---------------------------------------------------------------------- 2022-03-17T21:10:35.3420447Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:35.3848946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45293 2022-03-17T21:10:35.3979494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45294 2022-03-17T21:10:36.5105085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:36.5105619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:36.5106459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:36.5107143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:36.5213659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:36.6119193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:36.8031244Z skip (2.997s) 2022-03-17T21:10:36.8031501Z 2022-03-17T21:10:36.8032192Z ---------------------------------------------------------------------- 2022-03-17T21:10:36.8032720Z Ran 1 test in 2.997s 2022-03-17T21:10:36.8032879Z 2022-03-17T21:10:36.8032966Z OK (skipped=1) 2022-03-17T21:10:36.8033139Z 2022-03-17T21:10:36.8033258Z Generating XML reports... 2022-03-17T21:10:36.8076849Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211033.xml 2022-03-17T21:10:38.0837459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:38.0851260Z 2022-03-17T21:10:38.0851698Z Running tests... 2022-03-17T21:10:38.0852186Z ---------------------------------------------------------------------- 2022-03-17T21:10:39.6300503Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:39.6730106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45408 2022-03-17T21:10:39.6857034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45409 2022-03-17T21:10:40.8008080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:40.8008601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:40.8009371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:40.8010067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:40.8116343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:40.9022842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:41.0904178Z skip (3.005s) 2022-03-17T21:10:41.0904530Z 2022-03-17T21:10:41.0905037Z ---------------------------------------------------------------------- 2022-03-17T21:10:41.0905314Z Ran 1 test in 3.005s 2022-03-17T21:10:41.0905519Z 2022-03-17T21:10:41.0905590Z OK (skipped=1) 2022-03-17T21:10:41.0905746Z 2022-03-17T21:10:41.0905855Z Generating XML reports... 2022-03-17T21:10:41.0947739Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211038.xml 2022-03-17T21:10:42.3807447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:42.3821335Z 2022-03-17T21:10:42.3821495Z Running tests... 2022-03-17T21:10:42.3822429Z ---------------------------------------------------------------------- 2022-03-17T21:10:43.9328522Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:43.9756967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45523 2022-03-17T21:10:43.9887665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45524 2022-03-17T21:10:45.0932147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:45.0932601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:45.0933420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:45.0934112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:45.0940348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:45.0941492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:45.2934765Z skip (2.911s) 2022-03-17T21:10:45.2934956Z 2022-03-17T21:10:45.2935412Z ---------------------------------------------------------------------- 2022-03-17T21:10:45.2935723Z Ran 1 test in 2.911s 2022-03-17T21:10:45.2935874Z 2022-03-17T21:10:45.2935964Z OK (skipped=1) 2022-03-17T21:10:45.2936119Z 2022-03-17T21:10:45.2936246Z Generating XML reports... 2022-03-17T21:10:45.2980779Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211042.xml 2022-03-17T21:10:46.5751307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:46.5765260Z 2022-03-17T21:10:46.5765484Z Running tests... 2022-03-17T21:10:46.5766410Z ---------------------------------------------------------------------- 2022-03-17T21:10:48.1427505Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:48.1856098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45638 2022-03-17T21:10:48.1984908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45639 2022-03-17T21:10:49.3001035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:49.3001602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:49.3002350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:49.3003061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:49.3111695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:49.4015400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:49.6034807Z skip (3.026s) 2022-03-17T21:10:49.6035037Z 2022-03-17T21:10:49.6035444Z ---------------------------------------------------------------------- 2022-03-17T21:10:49.6035767Z Ran 1 test in 3.027s 2022-03-17T21:10:49.6035936Z 2022-03-17T21:10:49.6036041Z OK (skipped=1) 2022-03-17T21:10:49.6036179Z 2022-03-17T21:10:49.6036294Z Generating XML reports... 2022-03-17T21:10:49.6079894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211046.xml 2022-03-17T21:10:50.8927588Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:50.8943029Z 2022-03-17T21:10:50.8943201Z Running tests... 2022-03-17T21:10:50.8943611Z ---------------------------------------------------------------------- 2022-03-17T21:10:52.4442373Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:52.4873955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45753 2022-03-17T21:10:52.5003420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45754 2022-03-17T21:10:53.6232240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:53.6232837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:53.6233614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:53.6234318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:53.6240688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:53.6241853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:53.8049684Z ok (2.910s) 2022-03-17T21:10:53.8049874Z 2022-03-17T21:10:53.8050506Z ---------------------------------------------------------------------- 2022-03-17T21:10:53.8051057Z Ran 1 test in 2.911s 2022-03-17T21:10:53.8051276Z 2022-03-17T21:10:53.8051350Z OK 2022-03-17T21:10:53.8051487Z 2022-03-17T21:10:53.8051611Z Generating XML reports... 2022-03-17T21:10:53.8094848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211050.xml 2022-03-17T21:10:55.0909723Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:55.0923708Z 2022-03-17T21:10:55.0924192Z Running tests... 2022-03-17T21:10:55.0924743Z ---------------------------------------------------------------------- 2022-03-17T21:10:56.6383884Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:10:56.6813310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45872 2022-03-17T21:10:56.6942886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45873 2022-03-17T21:10:57.7498125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:10:57.7498669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:10:57.7499448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:57.7500121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:10:57.7608421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:10:57.8512562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:10:57.9990290Z ok (2.906s) 2022-03-17T21:10:57.9990513Z 2022-03-17T21:10:57.9991254Z ---------------------------------------------------------------------- 2022-03-17T21:10:57.9991527Z Ran 1 test in 2.907s 2022-03-17T21:10:57.9991648Z 2022-03-17T21:10:57.9991738Z OK 2022-03-17T21:10:57.9991872Z 2022-03-17T21:10:57.9992003Z Generating XML reports... 2022-03-17T21:10:58.0041748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211055.xml 2022-03-17T21:10:59.2922195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:10:59.2936511Z 2022-03-17T21:10:59.2936653Z Running tests... 2022-03-17T21:10:59.2937383Z ---------------------------------------------------------------------- 2022-03-17T21:10:59.2961089Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:10:59.2961325Z 2022-03-17T21:10:59.2961657Z ---------------------------------------------------------------------- 2022-03-17T21:10:59.2961995Z Ran 1 test in 0.003s 2022-03-17T21:10:59.2962163Z 2022-03-17T21:10:59.2962261Z OK (skipped=1) 2022-03-17T21:10:59.2962423Z 2022-03-17T21:10:59.2962501Z Generating XML reports... 2022-03-17T21:10:59.3007337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211059.xml 2022-03-17T21:11:00.4591895Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:00.4607265Z 2022-03-17T21:11:00.4607631Z Running tests... 2022-03-17T21:11:00.4608074Z ---------------------------------------------------------------------- 2022-03-17T21:11:02.0173815Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:02.0615455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46026 2022-03-17T21:11:02.0761321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46027 2022-03-17T21:11:03.1585172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:03.1585713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:03.1586504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:03.1587195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:03.1593272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:03.1594024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:03.3797051Z ok (2.919s) 2022-03-17T21:11:03.3797315Z 2022-03-17T21:11:03.3797722Z ---------------------------------------------------------------------- 2022-03-17T21:11:03.3798043Z Ran 1 test in 2.919s 2022-03-17T21:11:03.3798211Z 2022-03-17T21:11:03.3798313Z OK 2022-03-17T21:11:03.3798462Z 2022-03-17T21:11:03.3798580Z Generating XML reports... 2022-03-17T21:11:03.3841604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211100.xml 2022-03-17T21:11:04.6652347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:04.6667150Z 2022-03-17T21:11:04.6667293Z Running tests... 2022-03-17T21:11:04.6667716Z ---------------------------------------------------------------------- 2022-03-17T21:11:06.2099305Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:06.2529626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46145 2022-03-17T21:11:06.2658185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46146 2022-03-17T21:11:07.3641993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:07.3642533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:07.3643546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:07.3644227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:07.3750680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:07.4654990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:07.6708619Z ok (3.004s) 2022-03-17T21:11:07.6708923Z 2022-03-17T21:11:07.6709740Z ---------------------------------------------------------------------- 2022-03-17T21:11:07.6710136Z Ran 1 test in 3.004s 2022-03-17T21:11:07.6710315Z 2022-03-17T21:11:07.6710410Z OK 2022-03-17T21:11:07.6710528Z 2022-03-17T21:11:07.6710668Z Generating XML reports... 2022-03-17T21:11:07.6752569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211104.xml 2022-03-17T21:11:08.9710722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:08.9725195Z 2022-03-17T21:11:08.9725651Z Running tests... 2022-03-17T21:11:08.9726132Z ---------------------------------------------------------------------- 2022-03-17T21:11:08.9751042Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:11:08.9751308Z 2022-03-17T21:11:08.9751584Z ---------------------------------------------------------------------- 2022-03-17T21:11:08.9751923Z Ran 1 test in 0.003s 2022-03-17T21:11:08.9752107Z 2022-03-17T21:11:08.9753606Z OK (skipped=1) 2022-03-17T21:11:08.9753819Z 2022-03-17T21:11:08.9754265Z Generating XML reports... 2022-03-17T21:11:08.9796804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211108.xml 2022-03-17T21:11:10.1285831Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:10.1299590Z 2022-03-17T21:11:10.1299852Z Running tests... 2022-03-17T21:11:10.1300660Z ---------------------------------------------------------------------- 2022-03-17T21:11:10.1324185Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:11:10.1324460Z 2022-03-17T21:11:10.1324788Z ---------------------------------------------------------------------- 2022-03-17T21:11:10.1325098Z Ran 1 test in 0.003s 2022-03-17T21:11:10.1325265Z 2022-03-17T21:11:10.1325358Z OK (skipped=1) 2022-03-17T21:11:10.1325522Z 2022-03-17T21:11:10.1325672Z Generating XML reports... 2022-03-17T21:11:10.1370507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211110.xml 2022-03-17T21:11:11.2779185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:11.2793110Z 2022-03-17T21:11:11.2793347Z Running tests... 2022-03-17T21:11:11.2793775Z ---------------------------------------------------------------------- 2022-03-17T21:11:12.8471332Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:12.8903075Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46334 2022-03-17T21:11:12.9029964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46335 2022-03-17T21:11:13.9950743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:13.9951363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:13.9952157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:13.9952835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:13.9959862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:13.9960340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:14.2075836Z ok (2.928s) 2022-03-17T21:11:14.2076042Z 2022-03-17T21:11:14.2076426Z ---------------------------------------------------------------------- 2022-03-17T21:11:14.2076773Z Ran 1 test in 2.928s 2022-03-17T21:11:14.2076916Z 2022-03-17T21:11:14.2077012Z OK 2022-03-17T21:11:14.2077147Z 2022-03-17T21:11:14.2077283Z Generating XML reports... 2022-03-17T21:11:14.2119952Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211111.xml 2022-03-17T21:11:15.4953185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:15.4967368Z 2022-03-17T21:11:15.4967614Z Running tests... 2022-03-17T21:11:15.4968065Z ---------------------------------------------------------------------- 2022-03-17T21:11:17.0337946Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:17.0769004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46453 2022-03-17T21:11:17.0896193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46454 2022-03-17T21:11:18.2185387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:18.2185966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:18.2186725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:18.2187419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:18.2194114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:18.2194838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:18.3942631Z ok (2.897s) 2022-03-17T21:11:18.3942843Z 2022-03-17T21:11:18.3943217Z ---------------------------------------------------------------------- 2022-03-17T21:11:18.3943560Z Ran 1 test in 2.897s 2022-03-17T21:11:18.3943733Z 2022-03-17T21:11:18.3943904Z OK 2022-03-17T21:11:18.3944021Z 2022-03-17T21:11:18.3944155Z Generating XML reports... 2022-03-17T21:11:18.3987096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211115.xml 2022-03-17T21:11:19.6889448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:19.6904034Z 2022-03-17T21:11:19.6904477Z Running tests... 2022-03-17T21:11:19.6904947Z ---------------------------------------------------------------------- 2022-03-17T21:11:21.2402648Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:21.2833717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46572 2022-03-17T21:11:21.2962372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46573 2022-03-17T21:11:22.4272196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:22.4272742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:22.4273447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:22.4274126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:22.4280149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:22.4280894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:22.6009844Z ok (2.910s) 2022-03-17T21:11:22.6010078Z 2022-03-17T21:11:22.6010508Z ---------------------------------------------------------------------- 2022-03-17T21:11:22.6010827Z Ran 1 test in 2.910s 2022-03-17T21:11:22.6010982Z 2022-03-17T21:11:22.6011052Z OK 2022-03-17T21:11:22.6013688Z 2022-03-17T21:11:22.6014192Z Generating XML reports... 2022-03-17T21:11:22.6053946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211119.xml 2022-03-17T21:11:23.8936191Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:23.8950431Z 2022-03-17T21:11:23.8950819Z Running tests... 2022-03-17T21:11:23.8951324Z ---------------------------------------------------------------------- 2022-03-17T21:11:25.4351001Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:25.4777967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46687 2022-03-17T21:11:25.4906409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46688 2022-03-17T21:11:26.6037716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:26.6038396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:26.6039133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:26.6039822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:26.6146750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:26.7049839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:26.8957617Z ok (3.000s) 2022-03-17T21:11:26.8958096Z 2022-03-17T21:11:26.8958568Z ---------------------------------------------------------------------- 2022-03-17T21:11:26.8958947Z Ran 1 test in 3.001s 2022-03-17T21:11:26.8959130Z 2022-03-17T21:11:26.8959225Z OK 2022-03-17T21:11:26.8959344Z 2022-03-17T21:11:26.8959482Z Generating XML reports... 2022-03-17T21:11:26.9002287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211123.xml 2022-03-17T21:11:28.1771406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:28.1785368Z 2022-03-17T21:11:28.1785612Z Running tests... 2022-03-17T21:11:28.1786031Z ---------------------------------------------------------------------- 2022-03-17T21:11:28.1806136Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:11:28.1806440Z 2022-03-17T21:11:28.1806713Z ---------------------------------------------------------------------- 2022-03-17T21:11:28.1807021Z Ran 1 test in 0.002s 2022-03-17T21:11:28.1808224Z 2022-03-17T21:11:28.1808364Z OK (skipped=1) 2022-03-17T21:11:28.1808857Z 2022-03-17T21:11:28.1809141Z Generating XML reports... 2022-03-17T21:11:28.1851941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211128.xml 2022-03-17T21:11:29.3254935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:29.3269615Z 2022-03-17T21:11:29.3269859Z Running tests... 2022-03-17T21:11:29.3270310Z ---------------------------------------------------------------------- 2022-03-17T21:11:29.3291275Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:11:29.3291546Z 2022-03-17T21:11:29.3291829Z ---------------------------------------------------------------------- 2022-03-17T21:11:29.3292355Z Ran 1 test in 0.002s 2022-03-17T21:11:29.3292523Z 2022-03-17T21:11:29.3292640Z OK (skipped=1) 2022-03-17T21:11:29.3292796Z 2022-03-17T21:11:29.3292923Z Generating XML reports... 2022-03-17T21:11:29.3337122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211129.xml 2022-03-17T21:11:30.4327819Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:30.4342700Z 2022-03-17T21:11:30.4343129Z Running tests... 2022-03-17T21:11:30.4343627Z ---------------------------------------------------------------------- 2022-03-17T21:11:31.9879388Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:32.0309909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46876 2022-03-17T21:11:32.0442318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46877 2022-03-17T21:11:33.1894991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:33.1895554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:33.1896311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:33.1897007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:33.1903736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:33.1904219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:33.2113210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:11:33.2113717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:11:33.2114390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:11:33.2115159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:11:33.4490871Z ok (3.014s) 2022-03-17T21:11:33.4491062Z 2022-03-17T21:11:33.4491531Z ---------------------------------------------------------------------- 2022-03-17T21:11:33.4491813Z Ran 1 test in 3.015s 2022-03-17T21:11:33.4491960Z 2022-03-17T21:11:33.4492060Z OK 2022-03-17T21:11:33.4492175Z 2022-03-17T21:11:33.4492285Z Generating XML reports... 2022-03-17T21:11:33.4536135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211130.xml 2022-03-17T21:11:34.6862404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:34.6876999Z 2022-03-17T21:11:34.6877211Z Running tests... 2022-03-17T21:11:34.6877636Z ---------------------------------------------------------------------- 2022-03-17T21:11:36.2388501Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:36.2830400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47001 2022-03-17T21:11:36.2962553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47002 2022-03-17T21:11:37.4099869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:37.4100718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:37.4101505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:37.4102186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:37.4208204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:37.5115041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:37.7010868Z skip (3.013s) 2022-03-17T21:11:37.7011145Z 2022-03-17T21:11:37.7011557Z ---------------------------------------------------------------------- 2022-03-17T21:11:37.7011862Z Ran 1 test in 3.013s 2022-03-17T21:11:37.7011964Z 2022-03-17T21:11:37.7012074Z OK (skipped=1) 2022-03-17T21:11:37.7012231Z 2022-03-17T21:11:37.7012359Z Generating XML reports... 2022-03-17T21:11:37.7054999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211134.xml 2022-03-17T21:11:38.9566959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:38.9582046Z 2022-03-17T21:11:38.9582358Z Running tests... 2022-03-17T21:11:38.9582812Z ---------------------------------------------------------------------- 2022-03-17T21:11:40.5133386Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:40.5566380Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47116 2022-03-17T21:11:40.5697138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47117 2022-03-17T21:11:41.6509526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:41.6510086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:41.6510867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:41.6511564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:41.6518161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:41.6518645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:41.8743694Z ok (2.916s) 2022-03-17T21:11:41.8743920Z 2022-03-17T21:11:41.8744324Z ---------------------------------------------------------------------- 2022-03-17T21:11:41.8744698Z Ran 1 test in 2.916s 2022-03-17T21:11:41.8744867Z 2022-03-17T21:11:41.8744962Z OK 2022-03-17T21:11:41.8745087Z 2022-03-17T21:11:41.8745186Z Generating XML reports... 2022-03-17T21:11:41.8788971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211138.xml 2022-03-17T21:11:43.1633063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:43.1646991Z 2022-03-17T21:11:43.1647153Z Running tests... 2022-03-17T21:11:43.1647900Z ---------------------------------------------------------------------- 2022-03-17T21:11:44.7194396Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:44.7641481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47231 2022-03-17T21:11:44.7775336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47232 2022-03-17T21:11:45.8577450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:45.8577997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:45.8578696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:45.8579410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:45.8685768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:45.9588871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:46.1823932Z ok (3.017s) 2022-03-17T21:11:46.1824155Z 2022-03-17T21:11:46.1824538Z ---------------------------------------------------------------------- 2022-03-17T21:11:46.1824865Z Ran 1 test in 3.018s 2022-03-17T21:11:46.1825036Z 2022-03-17T21:11:46.1825462Z OK 2022-03-17T21:11:46.1825641Z 2022-03-17T21:11:46.1826058Z Generating XML reports... 2022-03-17T21:11:46.1868100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211143.xml 2022-03-17T21:11:47.4714587Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:47.4729140Z 2022-03-17T21:11:47.4729301Z Running tests... 2022-03-17T21:11:47.4729753Z ---------------------------------------------------------------------- 2022-03-17T21:11:49.0172377Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:49.0606790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47346 2022-03-17T21:11:49.0737703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47347 2022-03-17T21:11:50.1791749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:50.1792247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:50.1792976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:50.1793688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:50.1800886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:50.1801367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:50.3785163Z ok (2.905s) 2022-03-17T21:11:50.3785543Z 2022-03-17T21:11:50.3785971Z ---------------------------------------------------------------------- 2022-03-17T21:11:50.3786284Z Ran 1 test in 2.906s 2022-03-17T21:11:50.3786474Z 2022-03-17T21:11:50.3786558Z OK 2022-03-17T21:11:50.3786691Z 2022-03-17T21:11:50.3786825Z Generating XML reports... 2022-03-17T21:11:50.3828874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211147.xml 2022-03-17T21:11:51.6695938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:51.6710824Z 2022-03-17T21:11:51.6710978Z Running tests... 2022-03-17T21:11:51.6711764Z ---------------------------------------------------------------------- 2022-03-17T21:11:53.2093001Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:53.2532567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47461 2022-03-17T21:11:53.2664468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47462 2022-03-17T21:11:54.3284207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:54.3284763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:54.3285517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:54.3286228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:54.3392238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:54.4294547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:54.6712556Z ok (3.000s) 2022-03-17T21:11:54.6713070Z 2022-03-17T21:11:54.6713479Z ---------------------------------------------------------------------- 2022-03-17T21:11:54.6713803Z Ran 1 test in 3.000s 2022-03-17T21:11:54.6713951Z 2022-03-17T21:11:54.6714039Z OK 2022-03-17T21:11:54.6714176Z 2022-03-17T21:11:54.6714289Z Generating XML reports... 2022-03-17T21:11:54.6755555Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211151.xml 2022-03-17T21:11:55.9633452Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:11:55.9647697Z 2022-03-17T21:11:55.9648115Z Running tests... 2022-03-17T21:11:55.9648910Z ---------------------------------------------------------------------- 2022-03-17T21:11:57.4977845Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:11:57.5405449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47580 2022-03-17T21:11:57.5533312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47581 2022-03-17T21:11:58.6633786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:11:58.6634362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:11:58.6635139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:58.6635831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:11:58.6642481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:11:58.6643515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:11:58.8581379Z ok (2.893s) 2022-03-17T21:11:58.8581613Z 2022-03-17T21:11:58.8582039Z ---------------------------------------------------------------------- 2022-03-17T21:11:58.8582374Z Ran 1 test in 2.893s 2022-03-17T21:11:58.8582517Z 2022-03-17T21:11:58.8582595Z OK 2022-03-17T21:11:58.8582731Z 2022-03-17T21:11:58.8582869Z Generating XML reports... 2022-03-17T21:11:58.8625703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211155.xml 2022-03-17T21:12:00.1494136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:00.1508680Z 2022-03-17T21:12:00.1508947Z Running tests... 2022-03-17T21:12:00.1509390Z ---------------------------------------------------------------------- 2022-03-17T21:12:01.6877326Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:01.7330236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47699 2022-03-17T21:12:01.7468708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47700 2022-03-17T21:12:02.8304075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:02.8304621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:02.8305381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:02.8306060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:02.8312947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:02.8313850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:03.0516389Z ok (2.900s) 2022-03-17T21:12:03.0516656Z 2022-03-17T21:12:03.0517073Z ---------------------------------------------------------------------- 2022-03-17T21:12:03.0517661Z Ran 1 test in 2.901s 2022-03-17T21:12:03.0517833Z 2022-03-17T21:12:03.0517926Z OK 2022-03-17T21:12:03.0518064Z 2022-03-17T21:12:03.0518199Z Generating XML reports... 2022-03-17T21:12:03.0561810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211200.xml 2022-03-17T21:12:04.3476864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:04.3491109Z 2022-03-17T21:12:04.3491409Z Running tests... 2022-03-17T21:12:04.3491862Z ---------------------------------------------------------------------- 2022-03-17T21:12:04.3512110Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:12:04.3512412Z 2022-03-17T21:12:04.3512706Z ---------------------------------------------------------------------- 2022-03-17T21:12:04.3513043Z Ran 1 test in 0.002s 2022-03-17T21:12:04.3513190Z 2022-03-17T21:12:04.3513297Z OK (skipped=1) 2022-03-17T21:12:04.3513462Z 2022-03-17T21:12:04.3513586Z Generating XML reports... 2022-03-17T21:12:04.3557412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211204.xml 2022-03-17T21:12:05.5086980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:05.5101993Z 2022-03-17T21:12:05.5102290Z Running tests... 2022-03-17T21:12:05.5102745Z ---------------------------------------------------------------------- 2022-03-17T21:12:05.5124082Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:12:05.5124589Z 2022-03-17T21:12:05.5125062Z ---------------------------------------------------------------------- 2022-03-17T21:12:05.5125349Z Ran 1 test in 0.002s 2022-03-17T21:12:05.5125555Z 2022-03-17T21:12:05.5125659Z OK (skipped=1) 2022-03-17T21:12:05.5125764Z 2022-03-17T21:12:05.5125890Z Generating XML reports... 2022-03-17T21:12:05.5169800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211205.xml 2022-03-17T21:12:06.6580059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:06.6594707Z 2022-03-17T21:12:06.6595200Z Running tests... 2022-03-17T21:12:06.6595711Z ---------------------------------------------------------------------- 2022-03-17T21:12:06.6617900Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:12:06.6618144Z 2022-03-17T21:12:06.6618460Z ---------------------------------------------------------------------- 2022-03-17T21:12:06.6618821Z Ran 1 test in 0.002s 2022-03-17T21:12:06.6618961Z 2022-03-17T21:12:06.6619342Z OK (skipped=1) 2022-03-17T21:12:06.6619488Z 2022-03-17T21:12:06.6619841Z Generating XML reports... 2022-03-17T21:12:06.6663854Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211206.xml 2022-03-17T21:12:07.8160642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:07.8175193Z 2022-03-17T21:12:07.8175544Z Running tests... 2022-03-17T21:12:07.8176081Z ---------------------------------------------------------------------- 2022-03-17T21:12:09.3721202Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:09.4150430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47923 2022-03-17T21:12:09.4280694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47924 2022-03-17T21:12:10.5207793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:10.5208315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:10.5209136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:10.5210091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:10.5216219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:10.5216709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:10.7326794Z ok (2.915s) 2022-03-17T21:12:10.7327051Z 2022-03-17T21:12:10.7327664Z ---------------------------------------------------------------------- 2022-03-17T21:12:10.7328009Z Ran 1 test in 2.915s 2022-03-17T21:12:10.7328178Z 2022-03-17T21:12:10.7328282Z OK 2022-03-17T21:12:10.7328776Z 2022-03-17T21:12:10.7328938Z Generating XML reports... 2022-03-17T21:12:10.7370755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211207.xml 2022-03-17T21:12:12.0175407Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:12.0190086Z 2022-03-17T21:12:12.0190251Z Running tests... 2022-03-17T21:12:12.0191122Z ---------------------------------------------------------------------- 2022-03-17T21:12:13.5747227Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:13.6177835Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48042 2022-03-17T21:12:13.6309896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48043 2022-03-17T21:12:14.7028326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:14.7029362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:14.7030556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:14.7031270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:14.7136040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:14.8039521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:14.9356045Z ok (2.916s) 2022-03-17T21:12:14.9356283Z 2022-03-17T21:12:14.9356653Z ---------------------------------------------------------------------- 2022-03-17T21:12:14.9356994Z Ran 1 test in 2.917s 2022-03-17T21:12:14.9357165Z 2022-03-17T21:12:14.9357196Z OK 2022-03-17T21:12:14.9357312Z 2022-03-17T21:12:14.9357447Z Generating XML reports... 2022-03-17T21:12:14.9399627Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211212.xml 2022-03-17T21:12:16.2239897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:16.2254522Z 2022-03-17T21:12:16.2254964Z Running tests... 2022-03-17T21:12:16.2255461Z ---------------------------------------------------------------------- 2022-03-17T21:12:17.7741888Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:17.8172220Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48157 2022-03-17T21:12:17.8305247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48158 2022-03-17T21:12:18.9059528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:18.9060100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:18.9061163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:18.9061822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:18.9167313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:19.0070620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:19.2356284Z ok (3.010s) 2022-03-17T21:12:19.2356484Z 2022-03-17T21:12:19.2356979Z ---------------------------------------------------------------------- 2022-03-17T21:12:19.2357286Z Ran 1 test in 3.010s 2022-03-17T21:12:19.2357531Z 2022-03-17T21:12:19.2357562Z OK 2022-03-17T21:12:19.2357688Z 2022-03-17T21:12:19.2357809Z Generating XML reports... 2022-03-17T21:12:19.2402394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211216.xml 2022-03-17T21:12:20.5193727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:20.5208486Z 2022-03-17T21:12:20.5208940Z Running tests... 2022-03-17T21:12:20.5209464Z ---------------------------------------------------------------------- 2022-03-17T21:12:22.0424068Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:22.0853206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48276 2022-03-17T21:12:22.0979560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48277 2022-03-17T21:12:23.1703431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:23.1703959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:23.1704720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:23.1705396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:23.1813246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:23.2714425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:23.5028893Z ok (2.982s) 2022-03-17T21:12:23.5029094Z 2022-03-17T21:12:23.5029749Z ---------------------------------------------------------------------- 2022-03-17T21:12:23.5030074Z Ran 1 test in 2.982s 2022-03-17T21:12:23.5030164Z 2022-03-17T21:12:23.5030257Z OK 2022-03-17T21:12:23.5030394Z 2022-03-17T21:12:23.5030521Z Generating XML reports... 2022-03-17T21:12:23.5072270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211220.xml 2022-03-17T21:12:24.7982250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:24.7997396Z 2022-03-17T21:12:24.7997726Z Running tests... 2022-03-17T21:12:24.7998168Z ---------------------------------------------------------------------- 2022-03-17T21:12:26.3387275Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:26.3814399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48395 2022-03-17T21:12:26.3941862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48396 2022-03-17T21:12:27.4947834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:27.4948354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:27.4949075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:27.4949771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:27.4955907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:27.4956859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:27.6988986Z ok (2.899s) 2022-03-17T21:12:27.6989198Z 2022-03-17T21:12:27.6989866Z ---------------------------------------------------------------------- 2022-03-17T21:12:27.6990213Z Ran 1 test in 2.899s 2022-03-17T21:12:27.6990302Z 2022-03-17T21:12:27.6990372Z OK 2022-03-17T21:12:27.6990508Z 2022-03-17T21:12:27.6990682Z Generating XML reports... 2022-03-17T21:12:27.7032794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211224.xml 2022-03-17T21:12:28.9925095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:28.9939813Z 2022-03-17T21:12:28.9940374Z Running tests... 2022-03-17T21:12:28.9941164Z ---------------------------------------------------------------------- 2022-03-17T21:12:30.5430121Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:30.5868556Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48600 2022-03-17T21:12:30.5998800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48601 2022-03-17T21:12:31.6400927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:31.6401477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:31.6402257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:31.6402941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:31.6509362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:31.7415518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:33.3083035Z ok (4.314s) 2022-03-17T21:12:33.3083453Z 2022-03-17T21:12:33.3083961Z ---------------------------------------------------------------------- 2022-03-17T21:12:33.3084294Z Ran 1 test in 4.314s 2022-03-17T21:12:33.3084435Z 2022-03-17T21:12:33.3084463Z OK 2022-03-17T21:12:33.3084705Z 2022-03-17T21:12:33.3084923Z Generating XML reports... 2022-03-17T21:12:33.3127595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211228.xml 2022-03-17T21:12:34.6157938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:34.6171735Z 2022-03-17T21:12:34.6171968Z Running tests... 2022-03-17T21:12:34.6172532Z ---------------------------------------------------------------------- 2022-03-17T21:12:36.1586959Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:36.2018367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48807 2022-03-17T21:12:36.2148354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48808 2022-03-17T21:12:37.3295047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:37.3295876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:37.3296656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:37.3297627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:37.3405812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:37.4309451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:39.3227394Z ok (4.705s) 2022-03-17T21:12:39.3227595Z 2022-03-17T21:12:39.3228212Z ---------------------------------------------------------------------- 2022-03-17T21:12:39.3228549Z Ran 1 test in 4.705s 2022-03-17T21:12:39.3228683Z 2022-03-17T21:12:39.3228776Z OK 2022-03-17T21:12:39.3228894Z 2022-03-17T21:12:39.3229060Z Generating XML reports... 2022-03-17T21:12:39.3272999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211234.xml 2022-03-17T21:12:40.6270983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:40.6284771Z 2022-03-17T21:12:40.6285531Z Running tests... 2022-03-17T21:12:40.6286228Z ---------------------------------------------------------------------- 2022-03-17T21:12:42.1778713Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:42.2207063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48928 2022-03-17T21:12:42.2335180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48929 2022-03-17T21:12:43.3244774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:43.3245370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:43.3246135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:43.3246827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:43.3252940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:43.3253431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:43.5385695Z ok (2.910s) 2022-03-17T21:12:43.5385938Z 2022-03-17T21:12:43.5386336Z ---------------------------------------------------------------------- 2022-03-17T21:12:43.5386660Z Ran 1 test in 2.910s 2022-03-17T21:12:43.5386830Z 2022-03-17T21:12:43.5386921Z OK 2022-03-17T21:12:43.5387057Z 2022-03-17T21:12:43.5387191Z Generating XML reports... 2022-03-17T21:12:43.5429823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211240.xml 2022-03-17T21:12:44.8010768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:44.8027003Z 2022-03-17T21:12:44.8027509Z Running tests... 2022-03-17T21:12:44.8028027Z ---------------------------------------------------------------------- 2022-03-17T21:12:46.3446944Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:46.3879100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49047 2022-03-17T21:12:46.4008764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49048 2022-03-17T21:12:47.4593742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:47.4594745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:47.4596186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:47.4596990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:47.4701838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:47.5609402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:49.1090508Z ok (4.306s) 2022-03-17T21:12:49.1090722Z 2022-03-17T21:12:49.1091082Z ---------------------------------------------------------------------- 2022-03-17T21:12:49.1091667Z Ran 1 test in 4.306s 2022-03-17T21:12:49.1091836Z 2022-03-17T21:12:49.1091936Z OK 2022-03-17T21:12:49.1092070Z 2022-03-17T21:12:49.1092185Z Generating XML reports... 2022-03-17T21:12:49.1133588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211244.xml 2022-03-17T21:12:50.4131747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:50.4146300Z 2022-03-17T21:12:50.4146799Z Running tests... 2022-03-17T21:12:50.4147232Z ---------------------------------------------------------------------- 2022-03-17T21:12:51.9379962Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:51.9808260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49164 2022-03-17T21:12:51.9940624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49165 2022-03-17T21:12:53.0831372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:53.0831897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:53.0832698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:53.0833392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:53.0939485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:53.1844800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:54.7401912Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:12:54.7403511Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:12:55.1021543Z ok (4.687s) 2022-03-17T21:12:55.1021748Z 2022-03-17T21:12:55.1022172Z ---------------------------------------------------------------------- 2022-03-17T21:12:55.1022527Z Ran 1 test in 4.687s 2022-03-17T21:12:55.1022706Z 2022-03-17T21:12:55.1022812Z OK 2022-03-17T21:12:55.1022917Z 2022-03-17T21:12:55.1023063Z Generating XML reports... 2022-03-17T21:12:55.1068046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211250.xml 2022-03-17T21:12:56.3941747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:12:56.3955515Z 2022-03-17T21:12:56.3955956Z Running tests... 2022-03-17T21:12:56.3956464Z ---------------------------------------------------------------------- 2022-03-17T21:12:57.9192724Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:12:57.9614713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49285 2022-03-17T21:12:57.9742945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49286 2022-03-17T21:12:59.0909854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:12:59.0910386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:12:59.0911131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:59.0911826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:12:59.1019129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:12:59.1922584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:12:59.2132649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:12:59.2133291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:12:59.2133989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:59.2134690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:12:59.2342702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:12:59.2343572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:12:59.2344280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:12:59.2344967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:13:05.8901616Z ok (9.494s) 2022-03-17T21:13:05.8901967Z 2022-03-17T21:13:05.8902382Z ---------------------------------------------------------------------- 2022-03-17T21:13:05.8902667Z Ran 1 test in 9.495s 2022-03-17T21:13:05.8902832Z 2022-03-17T21:13:05.8902906Z OK 2022-03-17T21:13:05.8903042Z 2022-03-17T21:13:05.8903202Z Generating XML reports... 2022-03-17T21:13:05.8946527Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211256.xml 2022-03-17T21:13:07.1940031Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:07.1954421Z 2022-03-17T21:13:07.1955041Z Running tests... 2022-03-17T21:13:07.1955592Z ---------------------------------------------------------------------- 2022-03-17T21:13:08.7295785Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:13:08.7726053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49414 2022-03-17T21:13:08.7854194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49415 2022-03-17T21:13:09.8517015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:09.8517541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:09.8518291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:09.8518987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:09.8525738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:09.8526242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:09.8733140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:13:09.8733919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:13:09.8734626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:13:09.8735305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:13:09.8942005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:13:09.8942505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:13:09.8943909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:13:09.8944624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:13:16.6009748Z ok (9.405s) 2022-03-17T21:13:16.6009959Z 2022-03-17T21:13:16.6010357Z ---------------------------------------------------------------------- 2022-03-17T21:13:16.6010693Z Ran 1 test in 9.405s 2022-03-17T21:13:16.6010863Z 2022-03-17T21:13:16.6010961Z OK 2022-03-17T21:13:16.6011101Z 2022-03-17T21:13:16.6011214Z Generating XML reports... 2022-03-17T21:13:16.6053773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211307.xml 2022-03-17T21:13:16.9570976Z Running distributed tests for the gloo backend with file init_method 2022-03-17T21:13:16.9572414Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 21:13:16.956874] 2022-03-17T21:13:18.0212993Z 2022-03-17T21:13:18.0257004Z , <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_get_bucket_sizes>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_params_and_buffers>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]> 2022-03-17T21:13:18.0291362Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0291872Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0292421Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0292815Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0293257Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0293707Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0294193Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0294683Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0295248Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0295798Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0296331Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0296854Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0297377Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0297874Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0298346Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0298811Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0299270Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0299758Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0300162Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0301081Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0301546Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0302011Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0302420Z test_all_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0302795Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0303215Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0303619Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0304032Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0304457Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0304845Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0305222Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0305632Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0306027Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0306412Z test_all_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0306778Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0307177Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0307599Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0307980Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0308403Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0308823Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0309265Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0309687Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0310216Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0310640Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0311048Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0311473Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0311867Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0312301Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0312826Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0313225Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0313632Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0314031Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0314459Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0314861Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0315252Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0315659Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0316037Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0316423Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0316818Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0317197Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0317572Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0317925Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0318307Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0318694Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0319097Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0319485Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0319844Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0320218Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0320588Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0320971Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0321367Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0321752Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0322136Z test_all_to_all (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0322491Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0322868Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0323253Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0323628Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0324022Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0324394Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0324773Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0325152Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0325578Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0326009Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0326429Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0326874Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0327395Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0327820Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0328255Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0328666Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0329094Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0329532Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0330009Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0330471Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0330909Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0331362Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0331808Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0332210Z test_average_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0332598Z test_backend_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0332956Z test_backend_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0333319Z test_barrier (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0333663Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0334045Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0334437Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0334802Z test_barrier_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0335180Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0335558Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0335956Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0336349Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0336719Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0337115Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0337517Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0337926Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0338337Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0338732Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0339132Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0339523Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0339934Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0340821Z test_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0341186Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0341563Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0341933Z test_broadcast_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0342317Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0342710Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0343145Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0343648Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0344075Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0344579Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0344992Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0345402Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0345843Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0346288Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0346722Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0347136Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0347618Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0348035Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0348392Z test_ddp_device (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0348779Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0349160Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0349557Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0349965Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0350378Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0350812Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0351206Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0351660Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0352151Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0352688Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0353300Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0353893Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0354467Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0355056Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0355654Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0356226Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0356816Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0357355Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0357823Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0358263Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0358627Z test_ddp_inference (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0359022Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0359424Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0359800Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0360214Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0360700Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0361159Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0361622Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0362014Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0362400Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0362786Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0363304Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0363724Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0364136Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0364546Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0364957Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0365381Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0365779Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0366189Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0366601Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0366994Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0367389Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0367792Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0368245Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0368667Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0369028Z test_destroy_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0369425Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0369820Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0370231Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0370609Z test_gather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0370950Z test_gather_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0371315Z test_gather_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0371671Z test_gather_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0372050Z test_gather_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0372417Z test_gather_object (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0372780Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0373162Z test_get_backend (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0373505Z test_get_future (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0373859Z test_get_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0374217Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0374604Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0374988Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0375336Z test_irecv (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0375680Z test_isend (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0376037Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0376438Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0376846Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0377273Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0377794Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0378184Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0378596Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0379025Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0379426Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0379843Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0380702Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0381242Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0381658Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0382044Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0382437Z test_new_subgroups (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0382811Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0383269Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0383753Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0384197Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0384637Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0385075Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0385524Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0385948Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0386345Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0386763Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0387155Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0387541Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0387938Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0388315Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0388696Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0389053Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0389437Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0389822Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0390170Z test_reduce_max (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0390528Z test_reduce_min (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0390885Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0391261Z test_reduce_product (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0391606Z test_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0391970Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0392351Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0392715Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0393078Z test_scatter (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0393423Z test_scatter_checks (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0393796Z test_scatter_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0394166Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0394528Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0394914Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0395365Z test_scatter_group (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0395743Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0396097Z test_send_recv (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0396470Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0396885Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0397304Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0397729Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0398155Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0398562Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0398986Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0399379Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0399829Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0400220Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0400647Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0401055Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0401435Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0401841Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0402217Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0402597Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0403013Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0403436Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:18.0403872Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) 2022-03-17T21:13:19.0726908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:19.0741210Z 2022-03-17T21:13:19.0741487Z Running tests... 2022-03-17T21:13:19.0742163Z ---------------------------------------------------------------------- 2022-03-17T21:13:20.6117630Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:13:20.6546520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49578 2022-03-17T21:13:20.6674213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49579 2022-03-17T21:13:21.7883656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:21.7884336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:21.7885125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:21.7885818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:21.7992001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:21.8897716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:23.0844496Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:13:23.0845548Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:13:23.1868958Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:13:23.1870346Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:13:24.0932459Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:13:24.0933229Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:13:24.0934178Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:13:24.0935364Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:13:24.1079061Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:13:24.1079600Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:13:24.1080480Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:13:24.1081394Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:13:24.1226314Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:13:24.1226835Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy: 2022-03-17T21:13:24.1227663Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:13:24.1228559Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager: Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging. 2022-03-17T21:13:24.4765594Z ok (5.402s) 2022-03-17T21:13:24.4765806Z 2022-03-17T21:13:24.4766155Z ---------------------------------------------------------------------- 2022-03-17T21:13:24.4766491Z Ran 1 test in 5.402s 2022-03-17T21:13:24.4766654Z 2022-03-17T21:13:24.4766733Z OK 2022-03-17T21:13:24.4766832Z 2022-03-17T21:13:24.4766958Z Generating XML reports... 2022-03-17T21:13:24.4818894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211319.xml 2022-03-17T21:13:25.7861694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:25.7875960Z 2022-03-17T21:13:25.7876167Z Running tests... 2022-03-17T21:13:25.7876605Z ---------------------------------------------------------------------- 2022-03-17T21:13:25.7916689Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip (0.004s) 2022-03-17T21:13:25.7916971Z 2022-03-17T21:13:25.7917268Z ---------------------------------------------------------------------- 2022-03-17T21:13:25.7917580Z Ran 1 test in 0.004s 2022-03-17T21:13:25.7917726Z 2022-03-17T21:13:25.7917841Z OK (skipped=1) 2022-03-17T21:13:25.7917995Z 2022-03-17T21:13:25.7918117Z Generating XML reports... 2022-03-17T21:13:25.7962907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211325.xml 2022-03-17T21:13:26.9389907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:26.9403965Z 2022-03-17T21:13:26.9404219Z Running tests... 2022-03-17T21:13:26.9404623Z ---------------------------------------------------------------------- 2022-03-17T21:13:28.4790305Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:13:28.5219969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49731 2022-03-17T21:13:28.5350388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49732 2022-03-17T21:13:29.6086268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:29.6086804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:29.6087574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:29.6088469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:29.6197039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:29.7097831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:29.9397628Z ok (2.999s) 2022-03-17T21:13:29.9397845Z 2022-03-17T21:13:29.9398236Z ---------------------------------------------------------------------- 2022-03-17T21:13:29.9398573Z Ran 1 test in 2.999s 2022-03-17T21:13:29.9398740Z 2022-03-17T21:13:29.9398830Z OK 2022-03-17T21:13:29.9398964Z 2022-03-17T21:13:29.9399087Z Generating XML reports... 2022-03-17T21:13:29.9441385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211326.xml 2022-03-17T21:13:31.2297755Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:31.2313048Z 2022-03-17T21:13:31.2313542Z Running tests... 2022-03-17T21:13:31.2314122Z ---------------------------------------------------------------------- 2022-03-17T21:13:32.7819223Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:13:32.8249273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49846 2022-03-17T21:13:32.8381070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49847 2022-03-17T21:13:33.8986586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:33.8987162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:33.8987864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:33.8988541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:33.9094708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:34.0000557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:35.5810126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.5810620Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.5999934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.6000404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.6081130Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:35.6081846Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:35.6082848Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:35.6083872Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:35.6336544Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.6337022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.6714334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.6714981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.6899939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.6900708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.7139636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.7140104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.7326423Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.7326923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.7619055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.7619552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.7898455Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T21:13:35.7899019Z "You've set static_graph to be True, no need to set it again." 2022-03-17T21:13:35.7899564Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1762: UserWarning: You've set static_graph to be True, no need to set it again. 2022-03-17T21:13:35.7900149Z "You've set static_graph to be True, no need to set it again." 2022-03-17T21:13:35.8053140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.8053636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.8481924Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.8482416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.8910643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.8911103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.9286031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.9286523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.9471897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.9472386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.9762273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:35.9762732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.0137325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.0137792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.0323705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.0324347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.0564110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.0564602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.0749328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.0749809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.1038692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.1039288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.1473206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.1473691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.1897671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.1898147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.2325560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.2326038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:36.5469680Z ok (5.315s) 2022-03-17T21:13:36.5469926Z 2022-03-17T21:13:36.5470537Z ---------------------------------------------------------------------- 2022-03-17T21:13:36.5471059Z Ran 1 test in 5.316s 2022-03-17T21:13:36.5471163Z 2022-03-17T21:13:36.5471256Z OK 2022-03-17T21:13:36.5471392Z 2022-03-17T21:13:36.5471524Z Generating XML reports... 2022-03-17T21:13:36.5513330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211331.xml 2022-03-17T21:13:37.8505079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:37.8519585Z 2022-03-17T21:13:37.8519841Z Running tests... 2022-03-17T21:13:37.8520294Z ---------------------------------------------------------------------- 2022-03-17T21:13:39.3800289Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:13:39.4229506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49967 2022-03-17T21:13:39.4358821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49968 2022-03-17T21:13:40.5460740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:40.5461302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:40.5462107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:40.5462781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:40.5469713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:40.5470781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:40.5552380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:40.5553378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:40.5701964Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:40.5703801Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:40.5705908Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:40.5707455Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:40.7404364Z ok (2.888s) 2022-03-17T21:13:40.7404632Z 2022-03-17T21:13:40.7405289Z ---------------------------------------------------------------------- 2022-03-17T21:13:40.7405651Z Ran 1 test in 2.888s 2022-03-17T21:13:40.7405798Z 2022-03-17T21:13:40.7405889Z OK 2022-03-17T21:13:40.7406033Z 2022-03-17T21:13:40.7406169Z Generating XML reports... 2022-03-17T21:13:40.7447721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211337.xml 2022-03-17T21:13:42.0237056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:42.0251356Z 2022-03-17T21:13:42.0251772Z Running tests... 2022-03-17T21:13:42.0252274Z ---------------------------------------------------------------------- 2022-03-17T21:13:43.5848578Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:13:43.6282460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50086 2022-03-17T21:13:43.6414026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50087 2022-03-17T21:13:44.7387266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:44.7387795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:44.7388586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:44.7389275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:44.7495572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:44.8400105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:44.8484638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:44.8485149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:44.8640689Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:44.8641785Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:44.8642831Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:44.8643594Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:45.0462652Z ok (3.021s) 2022-03-17T21:13:45.0462891Z 2022-03-17T21:13:45.0463279Z ---------------------------------------------------------------------- 2022-03-17T21:13:45.0463594Z Ran 1 test in 3.021s 2022-03-17T21:13:45.0463763Z 2022-03-17T21:13:45.0463860Z OK 2022-03-17T21:13:45.0464263Z 2022-03-17T21:13:45.0464378Z Generating XML reports... 2022-03-17T21:13:45.0507006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211342.xml 2022-03-17T21:13:46.3387027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:46.3402115Z 2022-03-17T21:13:46.3402594Z Running tests... 2022-03-17T21:13:46.3403219Z ---------------------------------------------------------------------- 2022-03-17T21:13:47.8752520Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:13:47.9182026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50205 2022-03-17T21:13:47.9309883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50206 2022-03-17T21:13:49.0445097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:49.0445945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:49.0446625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:49.0447599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:49.0453507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:49.0454047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:51.0425977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.0426506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.0667071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.0667566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.0763565Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:51.0764445Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:51.0766065Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:51.0766770Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:51.0989714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.0990208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.1225928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.1226379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.2528547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.2528960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.2763212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.2763696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:51.8402848Z ok (5.500s) 2022-03-17T21:13:51.8403293Z 2022-03-17T21:13:51.8403706Z ---------------------------------------------------------------------- 2022-03-17T21:13:51.8404040Z Ran 1 test in 5.500s 2022-03-17T21:13:51.8404179Z 2022-03-17T21:13:51.8404242Z OK 2022-03-17T21:13:51.8404359Z 2022-03-17T21:13:51.8404491Z Generating XML reports... 2022-03-17T21:13:51.8446444Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211346.xml 2022-03-17T21:13:53.1372360Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:53.1386369Z 2022-03-17T21:13:53.1386678Z Running tests... 2022-03-17T21:13:53.1387394Z ---------------------------------------------------------------------- 2022-03-17T21:13:54.6725128Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:13:54.7148098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50326 2022-03-17T21:13:54.7278560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50327 2022-03-17T21:13:55.8383760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:13:55.8384305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:13:55.8385083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:55.8385787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:13:55.8392346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:13:55.8392971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:13:57.2164970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:57.2165557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:57.2328063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:57.2328780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:13:57.2398771Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:57.2399606Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:57.2410029Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:13:57.2410813Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:13:57.5349902Z ok (4.396s) 2022-03-17T21:13:57.5350076Z 2022-03-17T21:13:57.5350471Z ---------------------------------------------------------------------- 2022-03-17T21:13:57.5350822Z Ran 1 test in 4.396s 2022-03-17T21:13:57.5350986Z 2022-03-17T21:13:57.5351078Z OK 2022-03-17T21:13:57.5351195Z 2022-03-17T21:13:57.5351298Z Generating XML reports... 2022-03-17T21:13:57.5394819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211353.xml 2022-03-17T21:13:58.8513108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:13:58.8527625Z 2022-03-17T21:13:58.8528122Z Running tests... 2022-03-17T21:13:58.8528950Z ---------------------------------------------------------------------- 2022-03-17T21:14:00.4185212Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:00.4635345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50447 2022-03-17T21:14:00.4766181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50448 2022-03-17T21:14:01.5978086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:01.5978879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:01.5979695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:01.5980608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:01.6086777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:01.6992716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:03.0033135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:03.0033662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:03.0214409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:03.0214915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:03.0284890Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:14:03.0285724Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:14:03.0286757Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:14:03.0287523Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:14:03.2840114Z ok (4.431s) 2022-03-17T21:14:03.2840324Z 2022-03-17T21:14:03.2840710Z ---------------------------------------------------------------------- 2022-03-17T21:14:03.2841060Z Ran 1 test in 4.431s 2022-03-17T21:14:03.2841217Z 2022-03-17T21:14:03.2841314Z OK 2022-03-17T21:14:03.2841424Z 2022-03-17T21:14:03.2841565Z Generating XML reports... 2022-03-17T21:14:03.2891800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211358.xml 2022-03-17T21:14:04.5968463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:04.5983108Z 2022-03-17T21:14:04.5983254Z Running tests... 2022-03-17T21:14:04.5983974Z ---------------------------------------------------------------------- 2022-03-17T21:14:06.1592681Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:06.2023201Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50568 2022-03-17T21:14:06.2155144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50569 2022-03-17T21:14:07.2814855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:07.2815744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:07.2816763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:07.2817450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:07.2925226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:07.3829309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:08.6518959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:09.1230515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:09.1230904Z ok (4.524s) 2022-03-17T21:14:09.1231081Z 2022-03-17T21:14:09.1231461Z ---------------------------------------------------------------------- 2022-03-17T21:14:09.1231814Z Ran 1 test in 4.525s 2022-03-17T21:14:09.1231960Z 2022-03-17T21:14:09.1232054Z OK 2022-03-17T21:14:09.1232192Z 2022-03-17T21:14:09.1232320Z Generating XML reports... 2022-03-17T21:14:09.1274403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211404.xml 2022-03-17T21:14:10.4239787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:10.4253719Z 2022-03-17T21:14:10.4253870Z Running tests... 2022-03-17T21:14:10.4254847Z ---------------------------------------------------------------------- 2022-03-17T21:14:11.9823922Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:12.0251718Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50689 2022-03-17T21:14:12.0382710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50690 2022-03-17T21:14:13.1066141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:13.1066768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:13.1067676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:13.1068621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:13.1174501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:13.2080482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:15.1464055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:15.1464631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:15.1714585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:15.1715051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:15.1816639Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:14:15.1817470Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:14:15.1818490Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:14:15.1819511Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:14:15.6470370Z ok (5.221s) 2022-03-17T21:14:15.6470636Z 2022-03-17T21:14:15.6471030Z ---------------------------------------------------------------------- 2022-03-17T21:14:15.6471371Z Ran 1 test in 5.222s 2022-03-17T21:14:15.6471521Z 2022-03-17T21:14:15.6471616Z OK 2022-03-17T21:14:15.6473074Z 2022-03-17T21:14:15.6473480Z Generating XML reports... 2022-03-17T21:14:15.6515279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211410.xml 2022-03-17T21:14:16.9474786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:16.9488444Z 2022-03-17T21:14:16.9488580Z Running tests... 2022-03-17T21:14:16.9489283Z ---------------------------------------------------------------------- 2022-03-17T21:14:18.4800797Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:18.5221781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50810 2022-03-17T21:14:18.5348452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50811 2022-03-17T21:14:19.6202849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:19.6203382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:19.6204140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:19.6204838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:19.6210685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:19.6211573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:21.2574305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:21.2574874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:21.2773441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:21.2773920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:21.2853457Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:14:21.2854286Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:14:21.2855303Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:14:21.2856080Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:14:21.6426330Z ok (4.693s) 2022-03-17T21:14:21.6426541Z 2022-03-17T21:14:21.6426969Z ---------------------------------------------------------------------- 2022-03-17T21:14:21.6427294Z Ran 1 test in 4.694s 2022-03-17T21:14:21.6427464Z 2022-03-17T21:14:21.6427566Z OK 2022-03-17T21:14:21.6427707Z 2022-03-17T21:14:21.6427846Z Generating XML reports... 2022-03-17T21:14:21.6480485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211416.xml 2022-03-17T21:14:22.9410865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:22.9424934Z 2022-03-17T21:14:22.9425028Z Running tests... 2022-03-17T21:14:22.9425773Z ---------------------------------------------------------------------- 2022-03-17T21:14:24.4746472Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:24.5179954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50931 2022-03-17T21:14:24.5307570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50932 2022-03-17T21:14:25.6239025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:25.6239540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:25.6240358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:25.6241041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:25.6347337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:25.7252768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:27.0099713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:27.0100715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:27.0262577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:27.0263027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:27.0331236Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:14:27.0332297Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:14:27.0333533Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:14:27.0334341Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:14:27.3381707Z ok (4.395s) 2022-03-17T21:14:27.3381945Z 2022-03-17T21:14:27.3382361Z ---------------------------------------------------------------------- 2022-03-17T21:14:27.3382717Z Ran 1 test in 4.396s 2022-03-17T21:14:27.3382866Z 2022-03-17T21:14:27.3382969Z OK 2022-03-17T21:14:27.3383104Z 2022-03-17T21:14:27.3383247Z Generating XML reports... 2022-03-17T21:14:27.3428277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211422.xml 2022-03-17T21:14:28.6401682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:28.6415775Z 2022-03-17T21:14:28.6416030Z Running tests... 2022-03-17T21:14:28.6416486Z ---------------------------------------------------------------------- 2022-03-17T21:14:30.1819594Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:30.2249085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51052 2022-03-17T21:14:30.2377554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51053 2022-03-17T21:14:31.3004050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:31.3004684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:31.3005406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:31.3006079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:31.3013238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:31.3013744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:32.9044569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:32.9045141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:35.2485452Z ok (6.607s) 2022-03-17T21:14:35.2485738Z 2022-03-17T21:14:35.2486132Z ---------------------------------------------------------------------- 2022-03-17T21:14:35.2486481Z Ran 1 test in 6.607s 2022-03-17T21:14:35.2486627Z 2022-03-17T21:14:35.2486707Z OK 2022-03-17T21:14:35.2486854Z 2022-03-17T21:14:35.2486981Z Generating XML reports... 2022-03-17T21:14:35.2529315Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211428.xml 2022-03-17T21:14:36.5526765Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:36.5540989Z 2022-03-17T21:14:36.5541370Z Running tests... 2022-03-17T21:14:36.5541859Z ---------------------------------------------------------------------- 2022-03-17T21:14:38.0868901Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:38.1298605Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51173 2022-03-17T21:14:38.1429776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51174 2022-03-17T21:14:39.2662027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:39.2662524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:39.2663305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:39.2663989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:39.2671552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:39.2672268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:39.4475897Z ok (2.893s) 2022-03-17T21:14:39.4476056Z 2022-03-17T21:14:39.4476448Z ---------------------------------------------------------------------- 2022-03-17T21:14:39.4476838Z Ran 1 test in 2.893s 2022-03-17T21:14:39.4477011Z 2022-03-17T21:14:39.4477082Z OK 2022-03-17T21:14:39.4477340Z 2022-03-17T21:14:39.4477460Z Generating XML reports... 2022-03-17T21:14:39.4519840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211436.xml 2022-03-17T21:14:40.7314552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:40.7328525Z 2022-03-17T21:14:40.7329058Z Running tests... 2022-03-17T21:14:40.7329529Z ---------------------------------------------------------------------- 2022-03-17T21:14:42.2833312Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:42.3272403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51288 2022-03-17T21:14:42.3404244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51289 2022-03-17T21:14:43.4049772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:43.4050324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:43.4051084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:43.4052013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:43.4058736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:43.4059222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:45.0399144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:45.0399656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:45.1022350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:45.1022874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:14:45.4482162Z ok (4.715s) 2022-03-17T21:14:45.4482380Z 2022-03-17T21:14:45.4482775Z ---------------------------------------------------------------------- 2022-03-17T21:14:45.4483116Z Ran 1 test in 4.715s 2022-03-17T21:14:45.4483280Z 2022-03-17T21:14:45.4483394Z OK 2022-03-17T21:14:45.4483527Z 2022-03-17T21:14:45.4483654Z Generating XML reports... 2022-03-17T21:14:45.4525603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211440.xml 2022-03-17T21:14:46.7482799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:46.7497120Z 2022-03-17T21:14:46.7497456Z Running tests... 2022-03-17T21:14:46.7497955Z ---------------------------------------------------------------------- 2022-03-17T21:14:48.3002231Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:48.3430919Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51409 2022-03-17T21:14:48.3560334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51410 2022-03-17T21:14:49.4281369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:49.4282109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:49.4282889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:49.4283602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:49.4290150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:49.4290639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:51.0633362Z ok (4.313s) 2022-03-17T21:14:51.0633583Z 2022-03-17T21:14:51.0634009Z ---------------------------------------------------------------------- 2022-03-17T21:14:51.0634352Z Ran 1 test in 4.314s 2022-03-17T21:14:51.0634498Z 2022-03-17T21:14:51.0634569Z OK 2022-03-17T21:14:51.0634708Z 2022-03-17T21:14:51.0634867Z Generating XML reports... 2022-03-17T21:14:51.0677544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211446.xml 2022-03-17T21:14:52.3683488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:52.3697398Z 2022-03-17T21:14:52.3697825Z Running tests... 2022-03-17T21:14:52.3698310Z ---------------------------------------------------------------------- 2022-03-17T21:14:52.3720021Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:14:52.3720342Z 2022-03-17T21:14:52.3720630Z ---------------------------------------------------------------------- 2022-03-17T21:14:52.3720947Z Ran 1 test in 0.002s 2022-03-17T21:14:52.3721083Z 2022-03-17T21:14:52.3721194Z OK (skipped=1) 2022-03-17T21:14:52.3721351Z 2022-03-17T21:14:52.3721460Z Generating XML reports... 2022-03-17T21:14:52.3766690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211452.xml 2022-03-17T21:14:53.4727929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:53.4742462Z 2022-03-17T21:14:53.4742871Z Running tests... 2022-03-17T21:14:53.4743358Z ---------------------------------------------------------------------- 2022-03-17T21:14:53.4762508Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn) 2022-03-17T21:14:55.0127187Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:55.0558421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51561 2022-03-17T21:14:55.0688343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51562 2022-03-17T21:14:56.1500987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:14:56.1501782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:14:56.1502591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:56.1503289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:14:56.1509570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:14:56.1510057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:14:56.3737658Z ok (2.899s) 2022-03-17T21:14:56.3737895Z 2022-03-17T21:14:56.3738302Z ---------------------------------------------------------------------- 2022-03-17T21:14:56.3738635Z Ran 1 test in 2.899s 2022-03-17T21:14:56.3738784Z 2022-03-17T21:14:56.3738879Z OK 2022-03-17T21:14:56.3738994Z 2022-03-17T21:14:56.3739129Z Generating XML reports... 2022-03-17T21:14:56.3790996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211453.xml 2022-03-17T21:14:57.6572872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:14:57.6587190Z 2022-03-17T21:14:57.6587716Z Running tests... 2022-03-17T21:14:57.6588204Z ---------------------------------------------------------------------- 2022-03-17T21:14:57.6611333Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:14:59.2058572Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:14:59.2487311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51680 2022-03-17T21:14:59.2615761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51681 2022-03-17T21:15:00.3196645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:00.3197206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:00.3197952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:00.3198894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:00.3307434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:00.4208666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:00.4352392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:15:00.4352902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:15:00.6664644Z ok (3.007s) 2022-03-17T21:15:00.6665038Z 2022-03-17T21:15:00.6666009Z ---------------------------------------------------------------------- 2022-03-17T21:15:00.6666632Z Ran 1 test in 3.008s 2022-03-17T21:15:00.6666961Z 2022-03-17T21:15:00.6667131Z OK 2022-03-17T21:15:00.6667385Z 2022-03-17T21:15:00.6667629Z Generating XML reports... 2022-03-17T21:15:00.6710404Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211457.xml 2022-03-17T21:15:01.9526901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:01.9540749Z 2022-03-17T21:15:01.9541001Z Running tests... 2022-03-17T21:15:01.9541496Z ---------------------------------------------------------------------- 2022-03-17T21:15:01.9567262Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn) 2022-03-17T21:15:03.4974131Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:03.5403790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51799 2022-03-17T21:15:03.5532704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51800 2022-03-17T21:15:04.6706322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:04.6706888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:04.6707633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:04.6708341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:04.6714400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:04.6715113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:04.6846229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:15:04.6846721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:15:04.8583058Z ok (2.904s) 2022-03-17T21:15:04.8583319Z 2022-03-17T21:15:04.8583910Z ---------------------------------------------------------------------- 2022-03-17T21:15:04.8584257Z Ran 1 test in 2.904s 2022-03-17T21:15:04.8584374Z 2022-03-17T21:15:04.8584494Z OK 2022-03-17T21:15:04.8584642Z 2022-03-17T21:15:04.8584716Z Generating XML reports... 2022-03-17T21:15:04.8626944Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211501.xml 2022-03-17T21:15:06.1454715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:06.1469001Z 2022-03-17T21:15:06.1469487Z Running tests... 2022-03-17T21:15:06.1469999Z ---------------------------------------------------------------------- 2022-03-17T21:15:06.1488486Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn) 2022-03-17T21:15:07.6942676Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:07.7370339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51918 2022-03-17T21:15:07.7498703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51919 2022-03-17T21:15:08.8408256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:08.8408905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:08.8409637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:08.8410297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:08.8418142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:08.8418627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:09.0546638Z ok (2.907s) 2022-03-17T21:15:09.0546947Z 2022-03-17T21:15:09.0547337Z ---------------------------------------------------------------------- 2022-03-17T21:15:09.0547685Z Ran 1 test in 2.908s 2022-03-17T21:15:09.0547833Z 2022-03-17T21:15:09.0547929Z OK 2022-03-17T21:15:09.0549242Z 2022-03-17T21:15:09.0549639Z Generating XML reports... 2022-03-17T21:15:09.0591246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211506.xml 2022-03-17T21:15:10.3400111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:10.3414223Z 2022-03-17T21:15:10.3414370Z Running tests... 2022-03-17T21:15:10.3414831Z ---------------------------------------------------------------------- 2022-03-17T21:15:11.8912207Z test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:11.9338669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52037 2022-03-17T21:15:11.9466615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52038 2022-03-17T21:15:13.0551618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:13.0552210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:13.0552980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:13.0553654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:13.0559642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:13.0560391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:13.2517217Z ok (2.910s) 2022-03-17T21:15:13.2517442Z 2022-03-17T21:15:13.2517856Z ---------------------------------------------------------------------- 2022-03-17T21:15:13.2518203Z Ran 1 test in 2.910s 2022-03-17T21:15:13.2518370Z 2022-03-17T21:15:13.2518474Z OK 2022-03-17T21:15:13.2518611Z 2022-03-17T21:15:13.2518717Z Generating XML reports... 2022-03-17T21:15:13.2560901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211510.xml 2022-03-17T21:15:14.4862124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:14.4877616Z 2022-03-17T21:15:14.4877901Z Running tests... 2022-03-17T21:15:14.4878338Z ---------------------------------------------------------------------- 2022-03-17T21:15:16.0352101Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:16.0781568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52156 2022-03-17T21:15:16.0910187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52157 2022-03-17T21:15:17.1890895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:17.1891413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:17.1892178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:17.1892878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:17.1898889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:17.1899729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:17.3955931Z ok (2.907s) 2022-03-17T21:15:17.3956211Z 2022-03-17T21:15:17.3956544Z ---------------------------------------------------------------------- 2022-03-17T21:15:17.3956901Z Ran 1 test in 2.908s 2022-03-17T21:15:17.3957116Z 2022-03-17T21:15:17.3957237Z OK 2022-03-17T21:15:17.3957347Z 2022-03-17T21:15:17.3957441Z Generating XML reports... 2022-03-17T21:15:17.4006282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211514.xml 2022-03-17T21:15:18.6775263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:18.6791419Z 2022-03-17T21:15:18.6791887Z Running tests... 2022-03-17T21:15:18.6792324Z ---------------------------------------------------------------------- 2022-03-17T21:15:20.2319589Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:20.2749365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52275 2022-03-17T21:15:20.2878513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52276 2022-03-17T21:15:21.3879975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:21.3880553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:21.3881353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:21.3882031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:21.3888390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:21.3889145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:21.3996934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:15:21.3997481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:15:21.3998145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:15:21.3998821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:15:21.5925399Z ok (2.913s) 2022-03-17T21:15:21.5925601Z 2022-03-17T21:15:21.5926013Z ---------------------------------------------------------------------- 2022-03-17T21:15:21.5926333Z Ran 1 test in 2.913s 2022-03-17T21:15:21.5926473Z 2022-03-17T21:15:21.5926566Z OK 2022-03-17T21:15:21.5926701Z 2022-03-17T21:15:21.5926817Z Generating XML reports... 2022-03-17T21:15:21.5970272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211518.xml 2022-03-17T21:15:22.8810427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:22.8825072Z 2022-03-17T21:15:22.8825221Z Running tests... 2022-03-17T21:15:22.8825983Z ---------------------------------------------------------------------- 2022-03-17T21:15:24.4306940Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:24.4736891Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52400 2022-03-17T21:15:24.4864410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52401 2022-03-17T21:15:25.5883457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:25.5883993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:25.5885000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:25.5885729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:25.5892411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:25.5893356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:25.7910642Z skip (2.908s) 2022-03-17T21:15:25.7910867Z 2022-03-17T21:15:25.7911227Z ---------------------------------------------------------------------- 2022-03-17T21:15:25.7911578Z Ran 1 test in 2.909s 2022-03-17T21:15:25.7911716Z 2022-03-17T21:15:25.7911824Z OK (skipped=1) 2022-03-17T21:15:25.7911961Z 2022-03-17T21:15:25.7912089Z Generating XML reports... 2022-03-17T21:15:25.7955502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211522.xml 2022-03-17T21:15:27.0875336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:27.0889663Z 2022-03-17T21:15:27.0889818Z Running tests... 2022-03-17T21:15:27.0890303Z ---------------------------------------------------------------------- 2022-03-17T21:15:28.6422178Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:28.6854626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52515 2022-03-17T21:15:28.6990394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52516 2022-03-17T21:15:29.8072521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:29.8073085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:29.8073907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:29.8074608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:29.8180835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:29.9083913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:30.1038900Z ok (3.015s) 2022-03-17T21:15:30.1039145Z 2022-03-17T21:15:30.1039532Z ---------------------------------------------------------------------- 2022-03-17T21:15:30.1039901Z Ran 1 test in 3.015s 2022-03-17T21:15:30.1040059Z 2022-03-17T21:15:30.1040150Z OK 2022-03-17T21:15:30.1040283Z 2022-03-17T21:15:30.1040405Z Generating XML reports... 2022-03-17T21:15:30.1082606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211527.xml 2022-03-17T21:15:31.3970033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:31.3985146Z 2022-03-17T21:15:31.3985263Z Running tests... 2022-03-17T21:15:31.3985949Z ---------------------------------------------------------------------- 2022-03-17T21:15:32.9535294Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:32.9967350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52634 2022-03-17T21:15:33.0098063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52635 2022-03-17T21:15:34.1025979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:34.1026675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:34.1027467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:34.1028366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:34.1035728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:34.1036223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:34.3147000Z ok (2.916s) 2022-03-17T21:15:34.3147215Z 2022-03-17T21:15:34.3147803Z ---------------------------------------------------------------------- 2022-03-17T21:15:34.3148149Z Ran 1 test in 2.916s 2022-03-17T21:15:34.3148316Z 2022-03-17T21:15:34.3148397Z OK 2022-03-17T21:15:34.3148535Z 2022-03-17T21:15:34.3148671Z Generating XML reports... 2022-03-17T21:15:34.3190680Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211531.xml 2022-03-17T21:15:35.6095801Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:35.6110954Z 2022-03-17T21:15:35.6111180Z Running tests... 2022-03-17T21:15:35.6111606Z ---------------------------------------------------------------------- 2022-03-17T21:15:37.1624900Z test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:37.2060955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52753 2022-03-17T21:15:37.2194638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52754 2022-03-17T21:15:38.3078142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:38.3078782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:38.3079486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:38.3080187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:38.3086422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:38.3086928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:38.5242008Z ok (2.913s) 2022-03-17T21:15:38.5242256Z 2022-03-17T21:15:38.5242626Z ---------------------------------------------------------------------- 2022-03-17T21:15:38.5242968Z Ran 1 test in 2.913s 2022-03-17T21:15:38.5243079Z 2022-03-17T21:15:38.5243172Z OK 2022-03-17T21:15:38.5243307Z 2022-03-17T21:15:38.5243442Z Generating XML reports... 2022-03-17T21:15:38.5286886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211535.xml 2022-03-17T21:15:39.8198761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:39.8213767Z 2022-03-17T21:15:39.8214235Z Running tests... 2022-03-17T21:15:39.8214981Z ---------------------------------------------------------------------- 2022-03-17T21:15:39.8235940Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:39.8236338Z 2022-03-17T21:15:39.8236621Z ---------------------------------------------------------------------- 2022-03-17T21:15:39.8237270Z Ran 1 test in 0.002s 2022-03-17T21:15:39.8237437Z 2022-03-17T21:15:39.8237535Z OK (skipped=1) 2022-03-17T21:15:39.8237689Z 2022-03-17T21:15:39.8237816Z Generating XML reports... 2022-03-17T21:15:39.8283222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211539.xml 2022-03-17T21:15:40.9510192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:40.9525332Z 2022-03-17T21:15:40.9525752Z Running tests... 2022-03-17T21:15:40.9526258Z ---------------------------------------------------------------------- 2022-03-17T21:15:40.9548079Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:40.9548425Z 2022-03-17T21:15:40.9548715Z ---------------------------------------------------------------------- 2022-03-17T21:15:40.9549024Z Ran 1 test in 0.002s 2022-03-17T21:15:40.9549202Z 2022-03-17T21:15:40.9549263Z OK (skipped=1) 2022-03-17T21:15:40.9549415Z 2022-03-17T21:15:40.9549575Z Generating XML reports... 2022-03-17T21:15:40.9594545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211540.xml 2022-03-17T21:15:42.1174594Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:42.1189835Z 2022-03-17T21:15:42.1190220Z Running tests... 2022-03-17T21:15:42.1190707Z ---------------------------------------------------------------------- 2022-03-17T21:15:43.6540823Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:43.6969294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52942 2022-03-17T21:15:43.7100714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52943 2022-03-17T21:15:44.7904641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:44.7905230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:44.7906015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:44.7906690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:44.7913401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:44.7914306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:44.8121091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:15:44.8121605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:15:44.8122296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:15:44.8122968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:15:45.0147214Z ok (2.895s) 2022-03-17T21:15:45.0147434Z 2022-03-17T21:15:45.0147817Z ---------------------------------------------------------------------- 2022-03-17T21:15:45.0148162Z Ran 1 test in 2.896s 2022-03-17T21:15:45.0148318Z 2022-03-17T21:15:45.0148419Z OK 2022-03-17T21:15:45.0148557Z 2022-03-17T21:15:45.0148671Z Generating XML reports... 2022-03-17T21:15:45.0190512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211542.xml 2022-03-17T21:15:46.3084916Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:46.3099525Z 2022-03-17T21:15:46.3099840Z Running tests... 2022-03-17T21:15:46.3100642Z ---------------------------------------------------------------------- 2022-03-17T21:15:47.8567209Z test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:47.9011921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53067 2022-03-17T21:15:47.9139810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53068 2022-03-17T21:15:48.9998336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:48.9999280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:49.0000899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:49.0100204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:49.0107839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:49.1013936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:49.2186080Z skip (2.908s) 2022-03-17T21:15:49.2186261Z 2022-03-17T21:15:49.2186618Z ---------------------------------------------------------------------- 2022-03-17T21:15:49.2186968Z Ran 1 test in 2.909s 2022-03-17T21:15:49.2187133Z 2022-03-17T21:15:49.2187244Z OK (skipped=1) 2022-03-17T21:15:49.2187381Z 2022-03-17T21:15:49.2187507Z Generating XML reports... 2022-03-17T21:15:49.2230937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211546.xml 2022-03-17T21:15:50.4982647Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:50.4996733Z 2022-03-17T21:15:50.4997139Z Running tests... 2022-03-17T21:15:50.4997661Z ---------------------------------------------------------------------- 2022-03-17T21:15:50.5018817Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:50.5019129Z 2022-03-17T21:15:50.5019393Z ---------------------------------------------------------------------- 2022-03-17T21:15:50.5019738Z Ran 1 test in 0.002s 2022-03-17T21:15:50.5019906Z 2022-03-17T21:15:50.5020021Z OK (skipped=1) 2022-03-17T21:15:50.5020153Z 2022-03-17T21:15:50.5020587Z Generating XML reports... 2022-03-17T21:15:50.5064237Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211550.xml 2022-03-17T21:15:51.6456455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:51.6470804Z 2022-03-17T21:15:51.6471320Z Running tests... 2022-03-17T21:15:51.6471732Z ---------------------------------------------------------------------- 2022-03-17T21:15:51.6493625Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:15:51.6493953Z 2022-03-17T21:15:51.6494268Z ---------------------------------------------------------------------- 2022-03-17T21:15:51.6494602Z Ran 1 test in 0.002s 2022-03-17T21:15:51.6494767Z 2022-03-17T21:15:51.6494901Z OK (skipped=1) 2022-03-17T21:15:51.6495022Z 2022-03-17T21:15:51.6495130Z Generating XML reports... 2022-03-17T21:15:51.6540284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211551.xml 2022-03-17T21:15:52.7835954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:52.7849671Z 2022-03-17T21:15:52.7850170Z Running tests... 2022-03-17T21:15:52.7850610Z ---------------------------------------------------------------------- 2022-03-17T21:15:54.3082369Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:54.3506817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53252 2022-03-17T21:15:54.3635078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53253 2022-03-17T21:15:55.4196068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:55.4197052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:55.4198494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:55.4199575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:55.4304216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:55.5210226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:55.6681242Z ok (2.883s) 2022-03-17T21:15:55.6681387Z 2022-03-17T21:15:55.6681843Z ---------------------------------------------------------------------- 2022-03-17T21:15:55.6682162Z Ran 1 test in 2.883s 2022-03-17T21:15:55.6682355Z 2022-03-17T21:15:55.6682482Z OK 2022-03-17T21:15:55.6682579Z 2022-03-17T21:15:55.6682719Z Generating XML reports... 2022-03-17T21:15:55.6725049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211552.xml 2022-03-17T21:15:56.9546832Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:15:56.9561105Z 2022-03-17T21:15:56.9561458Z Running tests... 2022-03-17T21:15:56.9561942Z ---------------------------------------------------------------------- 2022-03-17T21:15:58.5078623Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:15:58.5507320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53367 2022-03-17T21:15:58.5635070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53368 2022-03-17T21:15:59.6867604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:15:59.6868345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:15:59.6869149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:59.6870100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:15:59.6976507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:15:59.7878935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:15:59.8192033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:15:59.8292008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:15:59.8292900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:15:59.8294108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:15:59.8542124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:15:59.8542642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:15:59.8543337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:15:59.8544006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:15:59.8768427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T21:15:59.8769156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T21:15:59.8769807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:15:59.8770484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:16:00.0685241Z ok (3.112s) 2022-03-17T21:16:00.0685483Z 2022-03-17T21:16:00.0685853Z ---------------------------------------------------------------------- 2022-03-17T21:16:00.0686183Z Ran 1 test in 3.112s 2022-03-17T21:16:00.0686571Z 2022-03-17T21:16:00.0686684Z OK 2022-03-17T21:16:00.0686821Z 2022-03-17T21:16:00.0686952Z Generating XML reports... 2022-03-17T21:16:00.0729552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211556.xml 2022-03-17T21:16:01.3633063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:01.3647879Z 2022-03-17T21:16:01.3648163Z Running tests... 2022-03-17T21:16:01.3648576Z ---------------------------------------------------------------------- 2022-03-17T21:16:02.9152293Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:02.9581677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53506 2022-03-17T21:16:02.9709591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53507 2022-03-17T21:16:04.0941731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:04.0942325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:04.0943060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:04.0943765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:04.0950607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:04.0951037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:04.1060203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:16:04.1060711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:16:04.1061616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:04.1062321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:04.2757196Z ok (2.911s) 2022-03-17T21:16:04.2757434Z 2022-03-17T21:16:04.2757835Z ---------------------------------------------------------------------- 2022-03-17T21:16:04.2758173Z Ran 1 test in 2.911s 2022-03-17T21:16:04.2758325Z 2022-03-17T21:16:04.2758426Z OK 2022-03-17T21:16:04.2758563Z 2022-03-17T21:16:04.2758693Z Generating XML reports... 2022-03-17T21:16:04.2801447Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211601.xml 2022-03-17T21:16:05.5569748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:05.5584203Z 2022-03-17T21:16:05.5584562Z Running tests... 2022-03-17T21:16:05.5585001Z ---------------------------------------------------------------------- 2022-03-17T21:16:07.1008978Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:07.1445707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53631 2022-03-17T21:16:07.1577605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53632 2022-03-17T21:16:08.2757050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:08.2757583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:08.2758360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:08.2759054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:08.2866559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:08.3770758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:08.3882209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:16:08.3882713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:16:08.3883368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:08.3884042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:08.5629595Z ok (3.004s) 2022-03-17T21:16:08.5629834Z 2022-03-17T21:16:08.5630252Z ---------------------------------------------------------------------- 2022-03-17T21:16:08.5630591Z Ran 1 test in 3.005s 2022-03-17T21:16:08.5630739Z 2022-03-17T21:16:08.5630896Z OK 2022-03-17T21:16:08.5630994Z 2022-03-17T21:16:08.5631129Z Generating XML reports... 2022-03-17T21:16:08.5674074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211605.xml 2022-03-17T21:16:09.8356974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:09.8370829Z 2022-03-17T21:16:09.8371141Z Running tests... 2022-03-17T21:16:09.8371535Z ---------------------------------------------------------------------- 2022-03-17T21:16:11.3596892Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:11.4020686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53756 2022-03-17T21:16:11.4148276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53757 2022-03-17T21:16:12.5295885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:12.5296435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:12.5297219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:12.5297924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:12.5403926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:12.6308039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:12.6516683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:16:12.6517214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:16:12.6517897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:12.6518595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:12.8195799Z ok (2.982s) 2022-03-17T21:16:12.8196070Z 2022-03-17T21:16:12.8196413Z ---------------------------------------------------------------------- 2022-03-17T21:16:12.8196755Z Ran 1 test in 2.982s 2022-03-17T21:16:12.8196896Z 2022-03-17T21:16:12.8196996Z OK 2022-03-17T21:16:12.8197132Z 2022-03-17T21:16:12.8197267Z Generating XML reports... 2022-03-17T21:16:12.8239476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211609.xml 2022-03-17T21:16:14.1108659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:14.1123151Z 2022-03-17T21:16:14.1123355Z Running tests... 2022-03-17T21:16:14.1124103Z ---------------------------------------------------------------------- 2022-03-17T21:16:15.6532976Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:15.6961297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53881 2022-03-17T21:16:15.7090165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53882 2022-03-17T21:16:16.8010667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:16.8011208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:16.8011970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:16.8012646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:16.8019207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:16.8019715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:16.8226985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:16:16.8227513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:16:16.8228183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:16.8228870Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:16:17.0136601Z ok (2.901s) 2022-03-17T21:16:17.0136845Z 2022-03-17T21:16:17.0137247Z ---------------------------------------------------------------------- 2022-03-17T21:16:17.0137576Z Ran 1 test in 2.901s 2022-03-17T21:16:17.0137717Z 2022-03-17T21:16:17.0137818Z OK 2022-03-17T21:16:17.0137935Z 2022-03-17T21:16:17.0138075Z Generating XML reports... 2022-03-17T21:16:17.0180828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211614.xml 2022-03-17T21:16:18.3063967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:18.3078292Z 2022-03-17T21:16:18.3078787Z Running tests... 2022-03-17T21:16:18.3079294Z ---------------------------------------------------------------------- 2022-03-17T21:16:19.8473467Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:19.8895606Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54006 2022-03-17T21:16:19.9022648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54007 2022-03-17T21:16:21.0534925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:21.0535413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:21.0536184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:21.0537165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:21.0645131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:21.1549892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:21.3070407Z skip (2.999s) 2022-03-17T21:16:21.3070629Z 2022-03-17T21:16:21.3071045Z ---------------------------------------------------------------------- 2022-03-17T21:16:21.3071371Z Ran 1 test in 2.999s 2022-03-17T21:16:21.3071512Z 2022-03-17T21:16:21.3071603Z OK (skipped=1) 2022-03-17T21:16:21.3072000Z 2022-03-17T21:16:21.3072144Z Generating XML reports... 2022-03-17T21:16:21.3114873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211618.xml 2022-03-17T21:16:22.5860096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:22.5874232Z 2022-03-17T21:16:22.5874520Z Running tests... 2022-03-17T21:16:22.5874957Z ---------------------------------------------------------------------- 2022-03-17T21:16:24.1345230Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:24.1770005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54121 2022-03-17T21:16:24.1898667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54122 2022-03-17T21:16:25.2962790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:25.2963320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:25.2964120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:25.2964792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:25.2974534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:25.2975722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:25.4944802Z skip (2.907s) 2022-03-17T21:16:25.4945004Z 2022-03-17T21:16:25.4945388Z ---------------------------------------------------------------------- 2022-03-17T21:16:25.4945719Z Ran 1 test in 2.907s 2022-03-17T21:16:25.4945896Z 2022-03-17T21:16:25.4945982Z OK (skipped=1) 2022-03-17T21:16:25.4946128Z 2022-03-17T21:16:25.4946203Z Generating XML reports... 2022-03-17T21:16:25.4988060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211622.xml 2022-03-17T21:16:26.7866028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:26.7880552Z 2022-03-17T21:16:26.7880701Z Running tests... 2022-03-17T21:16:26.7881579Z ---------------------------------------------------------------------- 2022-03-17T21:16:28.3308263Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:28.3735011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54236 2022-03-17T21:16:28.3867462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54237 2022-03-17T21:16:29.4924920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:29.4925416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:29.4926213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:29.4926874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:29.5032837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:29.5938586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:29.7916481Z skip (3.003s) 2022-03-17T21:16:29.7916695Z 2022-03-17T21:16:29.7917064Z ---------------------------------------------------------------------- 2022-03-17T21:16:29.7917421Z Ran 1 test in 3.004s 2022-03-17T21:16:29.7917576Z 2022-03-17T21:16:29.7917663Z OK (skipped=1) 2022-03-17T21:16:29.7917823Z 2022-03-17T21:16:29.7920134Z Generating XML reports... 2022-03-17T21:16:29.7960566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211626.xml 2022-03-17T21:16:31.0757665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:31.0773181Z 2022-03-17T21:16:31.0773661Z Running tests... 2022-03-17T21:16:31.0774130Z ---------------------------------------------------------------------- 2022-03-17T21:16:32.6310473Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:32.6740928Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54351 2022-03-17T21:16:32.6868275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54352 2022-03-17T21:16:33.7939325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:33.7940702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:33.7942117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:33.7943455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:33.7948910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:33.7949839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:33.9917040Z skip (2.914s) 2022-03-17T21:16:33.9917277Z 2022-03-17T21:16:33.9917705Z ---------------------------------------------------------------------- 2022-03-17T21:16:33.9918006Z Ran 1 test in 2.914s 2022-03-17T21:16:33.9918179Z 2022-03-17T21:16:33.9918290Z OK (skipped=1) 2022-03-17T21:16:33.9918433Z 2022-03-17T21:16:33.9918562Z Generating XML reports... 2022-03-17T21:16:33.9960411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211631.xml 2022-03-17T21:16:35.2899808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:35.2915557Z 2022-03-17T21:16:35.2916094Z Running tests... 2022-03-17T21:16:35.2916860Z ---------------------------------------------------------------------- 2022-03-17T21:16:36.8369789Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:36.8829400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54466 2022-03-17T21:16:36.8970915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54467 2022-03-17T21:16:37.9933837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:37.9934426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:37.9935237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:37.9935912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:37.9942297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:37.9943936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:38.2017471Z ok (2.910s) 2022-03-17T21:16:38.2017685Z 2022-03-17T21:16:38.2018088Z ---------------------------------------------------------------------- 2022-03-17T21:16:38.2018428Z Ran 1 test in 2.910s 2022-03-17T21:16:38.2018574Z 2022-03-17T21:16:38.2018667Z OK 2022-03-17T21:16:38.2018805Z 2022-03-17T21:16:38.2018940Z Generating XML reports... 2022-03-17T21:16:38.2061339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211635.xml 2022-03-17T21:16:39.5005953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:39.5020468Z 2022-03-17T21:16:39.5020723Z Running tests... 2022-03-17T21:16:39.5021477Z ---------------------------------------------------------------------- 2022-03-17T21:16:41.0592519Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:41.1041642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54585 2022-03-17T21:16:41.1178708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54586 2022-03-17T21:16:42.2519971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:42.2520479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:42.2521256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:42.2521948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:42.2528004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:42.2528462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:42.4226160Z ok (2.920s) 2022-03-17T21:16:42.4226730Z 2022-03-17T21:16:42.4227152Z ---------------------------------------------------------------------- 2022-03-17T21:16:42.4227498Z Ran 1 test in 2.920s 2022-03-17T21:16:42.4227651Z 2022-03-17T21:16:42.4227773Z OK 2022-03-17T21:16:42.4227963Z 2022-03-17T21:16:42.4228165Z Generating XML reports... 2022-03-17T21:16:42.4270178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211639.xml 2022-03-17T21:16:43.6679754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:43.6694340Z 2022-03-17T21:16:43.6694586Z Running tests... 2022-03-17T21:16:43.6695027Z ---------------------------------------------------------------------- 2022-03-17T21:16:45.2239281Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:45.2668696Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54700 2022-03-17T21:16:45.2798913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54701 2022-03-17T21:16:46.4169305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:46.4169858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:46.4170610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:46.4171300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:46.4178279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:46.4178961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:46.5846154Z ok (2.915s) 2022-03-17T21:16:46.5846406Z 2022-03-17T21:16:46.5846801Z ---------------------------------------------------------------------- 2022-03-17T21:16:46.5847124Z Ran 1 test in 2.915s 2022-03-17T21:16:46.5847298Z 2022-03-17T21:16:46.5847381Z OK 2022-03-17T21:16:46.5847494Z 2022-03-17T21:16:46.5847603Z Generating XML reports... 2022-03-17T21:16:46.5890353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211643.xml 2022-03-17T21:16:47.8701410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:47.8715562Z 2022-03-17T21:16:47.8715988Z Running tests... 2022-03-17T21:16:47.8716469Z ---------------------------------------------------------------------- 2022-03-17T21:16:49.4197123Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:49.4628167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54819 2022-03-17T21:16:49.4755004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54820 2022-03-17T21:16:50.5858296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:50.5858811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:50.5859595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:50.5860543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:50.5867123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:50.5868022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:50.7803568Z ok (2.908s) 2022-03-17T21:16:50.7803802Z 2022-03-17T21:16:50.7804164Z ---------------------------------------------------------------------- 2022-03-17T21:16:50.7804511Z Ran 1 test in 2.909s 2022-03-17T21:16:50.7804668Z 2022-03-17T21:16:50.7804742Z OK 2022-03-17T21:16:50.7804878Z 2022-03-17T21:16:50.7805016Z Generating XML reports... 2022-03-17T21:16:50.7847782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211647.xml 2022-03-17T21:16:52.0294748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:52.0311345Z 2022-03-17T21:16:52.0311749Z Running tests... 2022-03-17T21:16:52.0312270Z ---------------------------------------------------------------------- 2022-03-17T21:16:53.5646885Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:53.6077132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54938 2022-03-17T21:16:53.6208198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54939 2022-03-17T21:16:54.7281046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:54.7282015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:54.7283433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:54.7284815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:54.7290409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:54.7291287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:54.9256045Z ok (2.894s) 2022-03-17T21:16:54.9256754Z 2022-03-17T21:16:54.9257182Z ---------------------------------------------------------------------- 2022-03-17T21:16:54.9257451Z Ran 1 test in 2.894s 2022-03-17T21:16:54.9257618Z 2022-03-17T21:16:54.9257709Z OK 2022-03-17T21:16:54.9257847Z 2022-03-17T21:16:54.9257966Z Generating XML reports... 2022-03-17T21:16:54.9300104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211652.xml 2022-03-17T21:16:56.2226787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:16:56.2241123Z 2022-03-17T21:16:56.2241574Z Running tests... 2022-03-17T21:16:56.2242319Z ---------------------------------------------------------------------- 2022-03-17T21:16:57.7627588Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:16:57.8053416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55057 2022-03-17T21:16:57.8181041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55058 2022-03-17T21:16:58.9375869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:16:58.9376718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:16:58.9377512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:58.9378199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:16:58.9485280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:16:59.0386297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:16:59.2228201Z ok (2.998s) 2022-03-17T21:16:59.2228482Z 2022-03-17T21:16:59.2229166Z ---------------------------------------------------------------------- 2022-03-17T21:16:59.2229805Z Ran 1 test in 2.999s 2022-03-17T21:16:59.2229989Z 2022-03-17T21:16:59.2230084Z OK 2022-03-17T21:16:59.2230199Z 2022-03-17T21:16:59.2230326Z Generating XML reports... 2022-03-17T21:16:59.2272929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211656.xml 2022-03-17T21:17:00.5038050Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:00.5051333Z 2022-03-17T21:17:00.5051733Z Running tests... 2022-03-17T21:17:00.5052646Z ---------------------------------------------------------------------- 2022-03-17T21:17:02.0469102Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:02.0892877Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55172 2022-03-17T21:17:02.1021676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55173 2022-03-17T21:17:03.1598675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:03.1599352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:03.1600115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:03.1600805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:03.1707143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:03.2610764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:03.2820542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:17:03.2821766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:17:03.2823174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:03.2823884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:03.5073024Z ok (3.002s) 2022-03-17T21:17:03.5073236Z 2022-03-17T21:17:03.5073693Z ---------------------------------------------------------------------- 2022-03-17T21:17:03.5074006Z Ran 1 test in 3.002s 2022-03-17T21:17:03.5074165Z 2022-03-17T21:17:03.5074263Z OK 2022-03-17T21:17:03.5074408Z 2022-03-17T21:17:03.5074797Z Generating XML reports... 2022-03-17T21:17:03.5117531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211700.xml 2022-03-17T21:17:04.7968878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:04.7983174Z 2022-03-17T21:17:04.7983424Z Running tests... 2022-03-17T21:17:04.7983929Z ---------------------------------------------------------------------- 2022-03-17T21:17:06.3454056Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:06.3887886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55297 2022-03-17T21:17:06.4017640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55298 2022-03-17T21:17:07.4623795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:07.4624345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:07.4625152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:07.4625811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:07.4732004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:07.5635099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:07.5748487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:17:07.5749003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:17:07.5749741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:07.5750444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:07.8066413Z ok (3.008s) 2022-03-17T21:17:07.8066832Z 2022-03-17T21:17:07.8067213Z ---------------------------------------------------------------------- 2022-03-17T21:17:07.8067568Z Ran 1 test in 3.008s 2022-03-17T21:17:07.8067716Z 2022-03-17T21:17:07.8067811Z OK 2022-03-17T21:17:07.8067949Z 2022-03-17T21:17:07.8068064Z Generating XML reports... 2022-03-17T21:17:07.8110684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211704.xml 2022-03-17T21:17:09.0873302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:09.0887401Z 2022-03-17T21:17:09.0887889Z Running tests... 2022-03-17T21:17:09.0888305Z ---------------------------------------------------------------------- 2022-03-17T21:17:10.6392303Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:10.6829659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55422 2022-03-17T21:17:10.6963805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55423 2022-03-17T21:17:11.8048184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:11.8049057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:11.8049817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:11.8050495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:11.8158119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:11.9059627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:11.9176286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:17:11.9177012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:17:11.9177758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:11.9178445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:12.1012193Z ok (3.012s) 2022-03-17T21:17:12.1012450Z 2022-03-17T21:17:12.1013177Z ---------------------------------------------------------------------- 2022-03-17T21:17:12.1013842Z Ran 1 test in 3.012s 2022-03-17T21:17:12.1014042Z 2022-03-17T21:17:12.1014139Z OK 2022-03-17T21:17:12.1014283Z 2022-03-17T21:17:12.1014438Z Generating XML reports... 2022-03-17T21:17:12.1056810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211709.xml 2022-03-17T21:17:13.3947465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:13.3962558Z 2022-03-17T21:17:13.3962766Z Running tests... 2022-03-17T21:17:13.3963196Z ---------------------------------------------------------------------- 2022-03-17T21:17:14.9462052Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:14.9895569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55547 2022-03-17T21:17:15.0024979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55548 2022-03-17T21:17:16.0676922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:16.0677919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:16.0679208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:16.0680452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:16.0685478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:16.0687368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:16.0794890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:17:16.0795431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:17:16.0796128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:16.0796919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:17:16.3072130Z ok (2.911s) 2022-03-17T21:17:16.3072343Z 2022-03-17T21:17:16.3072752Z ---------------------------------------------------------------------- 2022-03-17T21:17:16.3073371Z Ran 1 test in 2.911s 2022-03-17T21:17:16.3073539Z 2022-03-17T21:17:16.3073640Z OK 2022-03-17T21:17:16.3073775Z 2022-03-17T21:17:16.3073889Z Generating XML reports... 2022-03-17T21:17:16.3117319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211713.xml 2022-03-17T21:17:17.5898888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:17.5914301Z 2022-03-17T21:17:17.5914510Z Running tests... 2022-03-17T21:17:17.5914950Z ---------------------------------------------------------------------- 2022-03-17T21:17:19.1481860Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:19.1911577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55672 2022-03-17T21:17:19.2043772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55673 2022-03-17T21:17:20.2925998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:20.2926533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:20.2927296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:20.2927989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:20.3036504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:20.3940101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:20.5088625Z skip (2.917s) 2022-03-17T21:17:20.5088812Z 2022-03-17T21:17:20.5089227Z ---------------------------------------------------------------------- 2022-03-17T21:17:20.5089538Z Ran 1 test in 2.917s 2022-03-17T21:17:20.5089675Z 2022-03-17T21:17:20.5089777Z OK (skipped=1) 2022-03-17T21:17:20.5089928Z 2022-03-17T21:17:20.5090038Z Generating XML reports... 2022-03-17T21:17:20.5132929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211717.xml 2022-03-17T21:17:21.8008435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:21.8023497Z 2022-03-17T21:17:21.8023732Z Running tests... 2022-03-17T21:17:21.8024154Z ---------------------------------------------------------------------- 2022-03-17T21:17:23.3428993Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:23.3861663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55787 2022-03-17T21:17:23.3991580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55788 2022-03-17T21:17:24.4654666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:24.4655213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:24.4656013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:24.4656703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:24.4765185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:24.5669566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:24.7037847Z skip (2.901s) 2022-03-17T21:17:24.7038064Z 2022-03-17T21:17:24.7038488Z ---------------------------------------------------------------------- 2022-03-17T21:17:24.7038808Z Ran 1 test in 2.901s 2022-03-17T21:17:24.7038960Z 2022-03-17T21:17:24.7039064Z OK (skipped=1) 2022-03-17T21:17:24.7039428Z 2022-03-17T21:17:24.7039560Z Generating XML reports... 2022-03-17T21:17:24.7082879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211721.xml 2022-03-17T21:17:25.9961943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:25.9976778Z 2022-03-17T21:17:25.9976910Z Running tests... 2022-03-17T21:17:25.9977388Z ---------------------------------------------------------------------- 2022-03-17T21:17:27.5519460Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:27.5951083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55902 2022-03-17T21:17:27.6080742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55903 2022-03-17T21:17:28.6987117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:28.6987882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:28.6988671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:28.6989604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:28.6995889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:28.6997016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:28.9126118Z skip (2.915s) 2022-03-17T21:17:28.9126533Z 2022-03-17T21:17:28.9127186Z ---------------------------------------------------------------------- 2022-03-17T21:17:28.9127776Z Ran 1 test in 2.915s 2022-03-17T21:17:28.9128079Z 2022-03-17T21:17:28.9128267Z OK (skipped=1) 2022-03-17T21:17:28.9128553Z 2022-03-17T21:17:28.9128798Z Generating XML reports... 2022-03-17T21:17:28.9172061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211725.xml 2022-03-17T21:17:30.1946239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:30.1961564Z 2022-03-17T21:17:30.1962112Z Running tests... 2022-03-17T21:17:30.1962556Z ---------------------------------------------------------------------- 2022-03-17T21:17:31.7342789Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:31.7772577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56017 2022-03-17T21:17:31.7904913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56018 2022-03-17T21:17:32.8744610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:32.8745140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:32.8745912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:32.8746587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:32.8752947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:32.8753877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:33.0949215Z skip (2.898s) 2022-03-17T21:17:33.0949440Z 2022-03-17T21:17:33.0949851Z ---------------------------------------------------------------------- 2022-03-17T21:17:33.0950175Z Ran 1 test in 2.899s 2022-03-17T21:17:33.0950347Z 2022-03-17T21:17:33.0950455Z OK (skipped=1) 2022-03-17T21:17:33.0950612Z 2022-03-17T21:17:33.0950722Z Generating XML reports... 2022-03-17T21:17:33.0993751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211730.xml 2022-03-17T21:17:34.3329722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:34.3343987Z 2022-03-17T21:17:34.3344307Z Running tests... 2022-03-17T21:17:34.3344931Z ---------------------------------------------------------------------- 2022-03-17T21:17:35.8717688Z test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:35.9147545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56132 2022-03-17T21:17:35.9277473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56133 2022-03-17T21:17:36.9879590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:36.9880127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:36.9880931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:36.9881626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:36.9990510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:37.0892051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:37.3324485Z ok (2.998s) 2022-03-17T21:17:37.3324688Z 2022-03-17T21:17:37.3325103Z ---------------------------------------------------------------------- 2022-03-17T21:17:37.3325449Z Ran 1 test in 2.998s 2022-03-17T21:17:37.3325601Z 2022-03-17T21:17:37.3325666Z OK 2022-03-17T21:17:37.3325801Z 2022-03-17T21:17:37.3325921Z Generating XML reports... 2022-03-17T21:17:37.3369270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211734.xml 2022-03-17T21:17:38.5994400Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:38.6008225Z 2022-03-17T21:17:38.6008736Z Running tests... 2022-03-17T21:17:38.6009376Z ---------------------------------------------------------------------- 2022-03-17T21:17:40.1278696Z test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:40.1701507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56251 2022-03-17T21:17:40.1827821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56252 2022-03-17T21:17:41.3113181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:41.3113887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:41.3114877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:41.3115605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:41.3223739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:41.4123619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:41.5874235Z ok (2.986s) 2022-03-17T21:17:41.5874499Z 2022-03-17T21:17:41.5874896Z ---------------------------------------------------------------------- 2022-03-17T21:17:41.5875231Z Ran 1 test in 2.987s 2022-03-17T21:17:41.5875404Z 2022-03-17T21:17:41.5875498Z OK 2022-03-17T21:17:41.5875641Z 2022-03-17T21:17:41.5875777Z Generating XML reports... 2022-03-17T21:17:41.5918776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211738.xml 2022-03-17T21:17:42.8644530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:42.8658726Z 2022-03-17T21:17:42.8659086Z Running tests... 2022-03-17T21:17:42.8659776Z ---------------------------------------------------------------------- 2022-03-17T21:17:44.3996335Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:44.4428134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56370 2022-03-17T21:17:44.4557408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56371 2022-03-17T21:17:45.5399833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:45.5400390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:45.5401162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:45.5401854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:45.5408403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:45.5408884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:47.5638274Z ok (4.698s) 2022-03-17T21:17:47.5638579Z 2022-03-17T21:17:47.5638954Z ---------------------------------------------------------------------- 2022-03-17T21:17:47.5639294Z Ran 1 test in 4.698s 2022-03-17T21:17:47.5639428Z 2022-03-17T21:17:47.5639531Z OK 2022-03-17T21:17:47.5639669Z 2022-03-17T21:17:47.5639821Z Generating XML reports... 2022-03-17T21:17:47.5684085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211742.xml 2022-03-17T21:17:48.8567282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:48.8581319Z 2022-03-17T21:17:48.8581642Z Running tests... 2022-03-17T21:17:48.8582097Z ---------------------------------------------------------------------- 2022-03-17T21:17:50.3970925Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:50.4399028Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56491 2022-03-17T21:17:50.4528993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56492 2022-03-17T21:17:51.5196173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:51.5196727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:51.5197498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:51.5198189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:51.5305139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:51.6210632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:53.5610242Z ok (4.702s) 2022-03-17T21:17:53.5610436Z 2022-03-17T21:17:53.5610831Z ---------------------------------------------------------------------- 2022-03-17T21:17:53.5611193Z Ran 1 test in 4.703s 2022-03-17T21:17:53.5611342Z 2022-03-17T21:17:53.5611434Z OK 2022-03-17T21:17:53.5611578Z 2022-03-17T21:17:53.5611694Z Generating XML reports... 2022-03-17T21:17:53.5664748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211748.xml 2022-03-17T21:17:54.8540030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:54.8554497Z 2022-03-17T21:17:54.8555167Z Running tests... 2022-03-17T21:17:54.8555624Z ---------------------------------------------------------------------- 2022-03-17T21:17:56.3975686Z test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:17:56.4403862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56612 2022-03-17T21:17:56.4532535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56613 2022-03-17T21:17:57.5457594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:17:57.5458368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:17:57.5459182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:57.5459871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:17:57.5466602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:17:57.5467092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:17:57.7579776Z ok (2.902s) 2022-03-17T21:17:57.7579932Z 2022-03-17T21:17:57.7580613Z ---------------------------------------------------------------------- 2022-03-17T21:17:57.7580966Z Ran 1 test in 2.902s 2022-03-17T21:17:57.7581134Z 2022-03-17T21:17:57.7581214Z OK 2022-03-17T21:17:57.7581360Z 2022-03-17T21:17:57.7581501Z Generating XML reports... 2022-03-17T21:17:57.7624887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211754.xml 2022-03-17T21:17:59.0464124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:17:59.0478145Z 2022-03-17T21:17:59.0478526Z Running tests... 2022-03-17T21:17:59.0479053Z ---------------------------------------------------------------------- 2022-03-17T21:18:00.5783609Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:18:00.6215392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56731 2022-03-17T21:18:00.6343337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56732 2022-03-17T21:18:01.7584542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:01.7585074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:01.7585878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:01.7586570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:01.7593306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:01.7594384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:03.4416018Z ok (4.393s) 2022-03-17T21:18:03.4416257Z 2022-03-17T21:18:03.4416620Z ---------------------------------------------------------------------- 2022-03-17T21:18:03.4416969Z Ran 1 test in 4.394s 2022-03-17T21:18:03.4417113Z 2022-03-17T21:18:03.4417186Z OK 2022-03-17T21:18:03.4417312Z 2022-03-17T21:18:03.4417442Z Generating XML reports... 2022-03-17T21:18:03.4459422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211759.xml 2022-03-17T21:18:04.7433992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:04.7448107Z 2022-03-17T21:18:04.7448557Z Running tests... 2022-03-17T21:18:04.7449066Z ---------------------------------------------------------------------- 2022-03-17T21:18:06.2687308Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:18:06.3113185Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56848 2022-03-17T21:18:06.3245053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56849 2022-03-17T21:18:07.3920833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:07.3921310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:07.3922325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:07.3923045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:07.3929191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:07.3929684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:07.6290164Z ok (2.884s) 2022-03-17T21:18:07.6290431Z 2022-03-17T21:18:07.6290836Z ---------------------------------------------------------------------- 2022-03-17T21:18:07.6291152Z Ran 1 test in 2.884s 2022-03-17T21:18:07.6291290Z 2022-03-17T21:18:07.6291378Z OK 2022-03-17T21:18:07.6291473Z 2022-03-17T21:18:07.6291588Z Generating XML reports... 2022-03-17T21:18:07.6333673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211804.xml 2022-03-17T21:18:08.9277016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:08.9291503Z 2022-03-17T21:18:08.9291749Z Running tests... 2022-03-17T21:18:08.9292203Z ---------------------------------------------------------------------- 2022-03-17T21:18:10.4808765Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:18:10.5241107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56967 2022-03-17T21:18:10.5367768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56968 2022-03-17T21:18:11.6356453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:11.6357058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:11.6357770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:11.6358457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:11.6467679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:11.7368061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:11.9416721Z ok (3.012s) 2022-03-17T21:18:11.9416944Z 2022-03-17T21:18:11.9417380Z ---------------------------------------------------------------------- 2022-03-17T21:18:11.9417747Z Ran 1 test in 3.012s 2022-03-17T21:18:11.9417886Z 2022-03-17T21:18:11.9417984Z OK 2022-03-17T21:18:11.9418122Z 2022-03-17T21:18:11.9418219Z Generating XML reports... 2022-03-17T21:18:11.9461524Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211808.xml 2022-03-17T21:18:13.2279819Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:13.2294552Z 2022-03-17T21:18:13.2295002Z Running tests... 2022-03-17T21:18:13.2295509Z ---------------------------------------------------------------------- 2022-03-17T21:18:14.7650546Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:18:14.8078101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57086 2022-03-17T21:18:14.8205462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57087 2022-03-17T21:18:15.9229037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:15.9229610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:15.9230354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:15.9231281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:15.9337529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:16.0241166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:16.2258031Z ok (2.996s) 2022-03-17T21:18:16.2258233Z 2022-03-17T21:18:16.2258632Z ---------------------------------------------------------------------- 2022-03-17T21:18:16.2258978Z Ran 1 test in 2.996s 2022-03-17T21:18:16.2259148Z 2022-03-17T21:18:16.2259233Z OK 2022-03-17T21:18:16.2259357Z 2022-03-17T21:18:16.2259424Z Generating XML reports... 2022-03-17T21:18:16.2302362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211813.xml 2022-03-17T21:18:17.5183334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:17.5199589Z 2022-03-17T21:18:17.5200063Z Running tests... 2022-03-17T21:18:17.5200551Z ---------------------------------------------------------------------- 2022-03-17T21:18:19.0775535Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:18:19.1208744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57205 2022-03-17T21:18:19.1340884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57206 2022-03-17T21:18:20.2350552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:20.2351085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:20.2351880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:20.2352499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:20.2358677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:20.2359963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:24.2455914Z ok (6.725s) 2022-03-17T21:18:24.2456136Z 2022-03-17T21:18:24.2456549Z ---------------------------------------------------------------------- 2022-03-17T21:18:24.2456871Z Ran 1 test in 6.726s 2022-03-17T21:18:24.2457038Z 2022-03-17T21:18:24.2457324Z OK 2022-03-17T21:18:24.2457470Z 2022-03-17T21:18:24.2457602Z Generating XML reports... 2022-03-17T21:18:24.2501226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211817.xml 2022-03-17T21:18:25.5229756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:25.5251010Z 2022-03-17T21:18:25.5251474Z Running tests... 2022-03-17T21:18:27.0792962Z ---------------------------------------------------------------------- 2022-03-17T21:18:27.0793519Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:18:27.1225419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57328 2022-03-17T21:18:27.1356399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57329 2022-03-17T21:18:28.2562956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:28.2563621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:28.2564408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:28.2565099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:28.2570837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:28.2571520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:32.2479821Z ok (6.723s) 2022-03-17T21:18:32.2480229Z 2022-03-17T21:18:32.2480626Z ---------------------------------------------------------------------- 2022-03-17T21:18:32.2480866Z Ran 1 test in 6.723s 2022-03-17T21:18:32.2481031Z 2022-03-17T21:18:32.2481121Z OK 2022-03-17T21:18:32.2481255Z 2022-03-17T21:18:32.2481388Z Generating XML reports... 2022-03-17T21:18:32.2522841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211825.xml 2022-03-17T21:18:33.5411618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:33.5425753Z 2022-03-17T21:18:33.5425923Z Running tests... 2022-03-17T21:18:33.5426644Z ---------------------------------------------------------------------- 2022-03-17T21:18:35.1054267Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:18:35.1480952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57451 2022-03-17T21:18:35.1611840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57452 2022-03-17T21:18:36.2955370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:18:36.2955898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:18:36.2956674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:36.2957364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:18:36.2964290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:18:36.2964814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:18:40.3728714Z ok (6.830s) 2022-03-17T21:18:40.3729111Z 2022-03-17T21:18:40.3729721Z ---------------------------------------------------------------------- 2022-03-17T21:18:40.3730316Z Ran 1 test in 6.830s 2022-03-17T21:18:40.3730668Z 2022-03-17T21:18:40.3730814Z OK 2022-03-17T21:18:40.3731075Z 2022-03-17T21:18:40.3731304Z Generating XML reports... 2022-03-17T21:18:40.3775037Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211833.xml 2022-03-17T21:18:41.6832805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:41.6847043Z 2022-03-17T21:18:41.6847508Z Running tests... 2022-03-17T21:18:41.6848008Z ---------------------------------------------------------------------- 2022-03-17T21:18:41.6867609Z test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:41.6868211Z 2022-03-17T21:18:41.6868577Z ---------------------------------------------------------------------- 2022-03-17T21:18:41.6868916Z Ran 1 test in 0.002s 2022-03-17T21:18:41.6869073Z 2022-03-17T21:18:41.6869179Z OK (skipped=1) 2022-03-17T21:18:41.6869336Z 2022-03-17T21:18:41.6869465Z Generating XML reports... 2022-03-17T21:18:41.6914270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211841.xml 2022-03-17T21:18:42.8390466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:42.8405058Z 2022-03-17T21:18:42.8405405Z Running tests... 2022-03-17T21:18:42.8405821Z ---------------------------------------------------------------------- 2022-03-17T21:18:42.8425714Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:42.8426187Z 2022-03-17T21:18:42.8426471Z ---------------------------------------------------------------------- 2022-03-17T21:18:42.8427093Z Ran 1 test in 0.002s 2022-03-17T21:18:42.8427276Z 2022-03-17T21:18:42.8427390Z OK (skipped=1) 2022-03-17T21:18:42.8427544Z 2022-03-17T21:18:42.8427684Z Generating XML reports... 2022-03-17T21:18:42.8472882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211842.xml 2022-03-17T21:18:43.9891712Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:43.9905957Z 2022-03-17T21:18:43.9906228Z Running tests... 2022-03-17T21:18:43.9906680Z ---------------------------------------------------------------------- 2022-03-17T21:18:43.9927166Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:43.9927449Z 2022-03-17T21:18:43.9927734Z ---------------------------------------------------------------------- 2022-03-17T21:18:43.9928022Z Ran 1 test in 0.002s 2022-03-17T21:18:43.9928185Z 2022-03-17T21:18:43.9928298Z OK (skipped=1) 2022-03-17T21:18:43.9928456Z 2022-03-17T21:18:43.9928616Z Generating XML reports... 2022-03-17T21:18:43.9972891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211843.xml 2022-03-17T21:18:45.1403088Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:45.1417465Z 2022-03-17T21:18:45.1417855Z Running tests... 2022-03-17T21:18:45.1418287Z ---------------------------------------------------------------------- 2022-03-17T21:18:45.1439001Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:45.1439434Z 2022-03-17T21:18:45.1439732Z ---------------------------------------------------------------------- 2022-03-17T21:18:45.1440027Z Ran 1 test in 0.002s 2022-03-17T21:18:45.1440190Z 2022-03-17T21:18:45.1440302Z OK (skipped=1) 2022-03-17T21:18:45.1440459Z 2022-03-17T21:18:45.1440564Z Generating XML reports... 2022-03-17T21:18:45.1484821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211845.xml 2022-03-17T21:18:46.3008813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:46.3023839Z 2022-03-17T21:18:46.3024235Z Running tests... 2022-03-17T21:18:46.3024685Z ---------------------------------------------------------------------- 2022-03-17T21:18:46.3044369Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:46.3044811Z 2022-03-17T21:18:46.3045111Z ---------------------------------------------------------------------- 2022-03-17T21:18:46.3045383Z Ran 1 test in 0.002s 2022-03-17T21:18:46.3045543Z 2022-03-17T21:18:46.3045656Z OK (skipped=1) 2022-03-17T21:18:46.3045811Z 2022-03-17T21:18:46.3045916Z Generating XML reports... 2022-03-17T21:18:46.3090138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211846.xml 2022-03-17T21:18:47.4562281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:47.4576885Z 2022-03-17T21:18:47.4577361Z Running tests... 2022-03-17T21:18:47.4577862Z ---------------------------------------------------------------------- 2022-03-17T21:18:47.4598571Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:47.4599081Z 2022-03-17T21:18:47.4599426Z ---------------------------------------------------------------------- 2022-03-17T21:18:47.4599757Z Ran 1 test in 0.002s 2022-03-17T21:18:47.4599907Z 2022-03-17T21:18:47.4600014Z OK (skipped=1) 2022-03-17T21:18:47.4600162Z 2022-03-17T21:18:47.4600295Z Generating XML reports... 2022-03-17T21:18:47.4644797Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211847.xml 2022-03-17T21:18:48.5543780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:48.5558583Z 2022-03-17T21:18:48.5559274Z Running tests... 2022-03-17T21:18:48.5559749Z ---------------------------------------------------------------------- 2022-03-17T21:18:48.5578742Z test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:48.5579152Z 2022-03-17T21:18:48.5579754Z ---------------------------------------------------------------------- 2022-03-17T21:18:48.5580161Z Ran 1 test in 0.002s 2022-03-17T21:18:48.5580543Z 2022-03-17T21:18:48.5580707Z OK (skipped=1) 2022-03-17T21:18:48.5580777Z 2022-03-17T21:18:48.5580904Z Generating XML reports... 2022-03-17T21:18:48.5625400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211848.xml 2022-03-17T21:18:49.7005711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:49.7019588Z 2022-03-17T21:18:49.7019937Z Running tests... 2022-03-17T21:18:49.7020471Z ---------------------------------------------------------------------- 2022-03-17T21:18:49.7040788Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:49.7041133Z 2022-03-17T21:18:49.7041412Z ---------------------------------------------------------------------- 2022-03-17T21:18:49.7041734Z Ran 1 test in 0.002s 2022-03-17T21:18:49.7041895Z 2022-03-17T21:18:49.7041996Z OK (skipped=1) 2022-03-17T21:18:49.7042130Z 2022-03-17T21:18:49.7042250Z Generating XML reports... 2022-03-17T21:18:49.7086752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211849.xml 2022-03-17T21:18:50.8652273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:50.8666852Z 2022-03-17T21:18:50.8667300Z Running tests... 2022-03-17T21:18:50.8667784Z ---------------------------------------------------------------------- 2022-03-17T21:18:50.8686935Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:50.8687226Z 2022-03-17T21:18:50.8687605Z ---------------------------------------------------------------------- 2022-03-17T21:18:50.8687896Z Ran 1 test in 0.002s 2022-03-17T21:18:50.8688056Z 2022-03-17T21:18:50.8688149Z OK (skipped=1) 2022-03-17T21:18:50.8688257Z 2022-03-17T21:18:50.8688390Z Generating XML reports... 2022-03-17T21:18:50.8732592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211850.xml 2022-03-17T21:18:52.0127894Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:52.0141213Z 2022-03-17T21:18:52.0141367Z Running tests... 2022-03-17T21:18:52.0142309Z ---------------------------------------------------------------------- 2022-03-17T21:18:52.0161621Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:52.0162092Z 2022-03-17T21:18:52.0162505Z ---------------------------------------------------------------------- 2022-03-17T21:18:52.0162816Z Ran 1 test in 0.002s 2022-03-17T21:18:52.0162978Z 2022-03-17T21:18:52.0163070Z OK (skipped=1) 2022-03-17T21:18:52.0163224Z 2022-03-17T21:18:52.0163352Z Generating XML reports... 2022-03-17T21:18:52.0207907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211852.xml 2022-03-17T21:18:53.1671754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:53.1685360Z 2022-03-17T21:18:53.1685622Z Running tests... 2022-03-17T21:18:53.1686074Z ---------------------------------------------------------------------- 2022-03-17T21:18:53.1707715Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:53.1707995Z 2022-03-17T21:18:53.1708351Z ---------------------------------------------------------------------- 2022-03-17T21:18:53.1708637Z Ran 1 test in 0.002s 2022-03-17T21:18:53.1708798Z 2022-03-17T21:18:53.1709122Z OK (skipped=1) 2022-03-17T21:18:53.1709313Z 2022-03-17T21:18:53.1709441Z Generating XML reports... 2022-03-17T21:18:53.1754458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211853.xml 2022-03-17T21:18:54.3062892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:54.3077847Z 2022-03-17T21:18:54.3078282Z Running tests... 2022-03-17T21:18:54.3078778Z ---------------------------------------------------------------------- 2022-03-17T21:18:54.3099793Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:54.3100213Z 2022-03-17T21:18:54.3101227Z ---------------------------------------------------------------------- 2022-03-17T21:18:54.3101761Z Ran 1 test in 0.002s 2022-03-17T21:18:54.3101930Z 2022-03-17T21:18:54.3102023Z OK (skipped=1) 2022-03-17T21:18:54.3102179Z 2022-03-17T21:18:54.3102321Z Generating XML reports... 2022-03-17T21:18:54.3147205Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211854.xml 2022-03-17T21:18:55.4578683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:55.4593148Z 2022-03-17T21:18:55.4593455Z Running tests... 2022-03-17T21:18:55.4594393Z ---------------------------------------------------------------------- 2022-03-17T21:18:55.4614288Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:55.4614822Z 2022-03-17T21:18:55.4615330Z ---------------------------------------------------------------------- 2022-03-17T21:18:55.4615908Z Ran 1 test in 0.002s 2022-03-17T21:18:55.4616157Z 2022-03-17T21:18:55.4616380Z OK (skipped=1) 2022-03-17T21:18:55.4616709Z 2022-03-17T21:18:55.4617017Z Generating XML reports... 2022-03-17T21:18:55.4661427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211855.xml 2022-03-17T21:18:56.6080589Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:56.6094849Z 2022-03-17T21:18:56.6095115Z Running tests... 2022-03-17T21:18:56.6095520Z ---------------------------------------------------------------------- 2022-03-17T21:18:56.6117383Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:56.6117674Z 2022-03-17T21:18:56.6117971Z ---------------------------------------------------------------------- 2022-03-17T21:18:56.6118280Z Ran 1 test in 0.002s 2022-03-17T21:18:56.6118445Z 2022-03-17T21:18:56.6118534Z OK (skipped=1) 2022-03-17T21:18:56.6118715Z 2022-03-17T21:18:56.6118741Z Generating XML reports... 2022-03-17T21:18:56.6163989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211856.xml 2022-03-17T21:18:57.7659880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:57.7674511Z 2022-03-17T21:18:57.7674894Z Running tests... 2022-03-17T21:18:57.7675380Z ---------------------------------------------------------------------- 2022-03-17T21:18:57.7695393Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:57.7695675Z 2022-03-17T21:18:57.7695981Z ---------------------------------------------------------------------- 2022-03-17T21:18:57.7696316Z Ran 1 test in 0.002s 2022-03-17T21:18:57.7696472Z 2022-03-17T21:18:57.7696562Z OK (skipped=1) 2022-03-17T21:18:57.7696673Z 2022-03-17T21:18:57.7696767Z Generating XML reports... 2022-03-17T21:18:57.7742116Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211857.xml 2022-03-17T21:18:58.9105781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:18:58.9120537Z 2022-03-17T21:18:58.9120710Z Running tests... 2022-03-17T21:18:58.9121137Z ---------------------------------------------------------------------- 2022-03-17T21:18:58.9142279Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:18:58.9142668Z 2022-03-17T21:18:58.9142976Z ---------------------------------------------------------------------- 2022-03-17T21:18:58.9143336Z Ran 1 test in 0.002s 2022-03-17T21:18:58.9143471Z 2022-03-17T21:18:58.9143614Z OK (skipped=1) 2022-03-17T21:18:58.9143743Z 2022-03-17T21:18:58.9143867Z Generating XML reports... 2022-03-17T21:18:58.9189502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211858.xml 2022-03-17T21:19:00.0625789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:00.0640077Z 2022-03-17T21:19:00.0640506Z Running tests... 2022-03-17T21:19:00.0641030Z ---------------------------------------------------------------------- 2022-03-17T21:19:00.0660970Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:00.0661291Z 2022-03-17T21:19:00.0661563Z ---------------------------------------------------------------------- 2022-03-17T21:19:00.0661875Z Ran 1 test in 0.002s 2022-03-17T21:19:00.0662066Z 2022-03-17T21:19:00.0662364Z OK (skipped=1) 2022-03-17T21:19:00.0662527Z 2022-03-17T21:19:00.0662672Z Generating XML reports... 2022-03-17T21:19:00.0708653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211900.xml 2022-03-17T21:19:01.1727048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:01.1741783Z 2022-03-17T21:19:01.1742172Z Running tests... 2022-03-17T21:19:01.1742594Z ---------------------------------------------------------------------- 2022-03-17T21:19:01.1763652Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:01.1763960Z 2022-03-17T21:19:01.1764231Z ---------------------------------------------------------------------- 2022-03-17T21:19:01.1764543Z Ran 1 test in 0.002s 2022-03-17T21:19:01.1764727Z 2022-03-17T21:19:01.1764831Z OK (skipped=1) 2022-03-17T21:19:01.1764987Z 2022-03-17T21:19:01.1765090Z Generating XML reports... 2022-03-17T21:19:01.1811086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211901.xml 2022-03-17T21:19:02.3320934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:02.3335757Z 2022-03-17T21:19:02.3336275Z Running tests... 2022-03-17T21:19:02.3336911Z ---------------------------------------------------------------------- 2022-03-17T21:19:02.3358776Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:02.3359072Z 2022-03-17T21:19:02.3359527Z ---------------------------------------------------------------------- 2022-03-17T21:19:02.3359950Z Ran 1 test in 0.002s 2022-03-17T21:19:02.3360112Z 2022-03-17T21:19:02.3360212Z OK (skipped=1) 2022-03-17T21:19:02.3360367Z 2022-03-17T21:19:02.3360972Z Generating XML reports... 2022-03-17T21:19:02.3405838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211902.xml 2022-03-17T21:19:03.4850755Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:03.4865436Z 2022-03-17T21:19:03.4865720Z Running tests... 2022-03-17T21:19:03.4866163Z ---------------------------------------------------------------------- 2022-03-17T21:19:03.4887776Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:03.4888075Z 2022-03-17T21:19:03.4888688Z ---------------------------------------------------------------------- 2022-03-17T21:19:03.4888979Z Ran 1 test in 0.002s 2022-03-17T21:19:03.4889147Z 2022-03-17T21:19:03.4889315Z OK (skipped=1) 2022-03-17T21:19:03.4889466Z 2022-03-17T21:19:03.4889554Z Generating XML reports... 2022-03-17T21:19:03.4934478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211903.xml 2022-03-17T21:19:04.6381001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:04.6395233Z 2022-03-17T21:19:04.6395466Z Running tests... 2022-03-17T21:19:04.6395930Z ---------------------------------------------------------------------- 2022-03-17T21:19:04.6415801Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:04.6416079Z 2022-03-17T21:19:04.6416381Z ---------------------------------------------------------------------- 2022-03-17T21:19:04.6416752Z Ran 1 test in 0.002s 2022-03-17T21:19:04.6416913Z 2022-03-17T21:19:04.6417043Z OK (skipped=1) 2022-03-17T21:19:04.6417193Z 2022-03-17T21:19:04.6417283Z Generating XML reports... 2022-03-17T21:19:04.6462318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211904.xml 2022-03-17T21:19:05.7873415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:05.7887317Z 2022-03-17T21:19:05.7887747Z Running tests... 2022-03-17T21:19:05.7888426Z ---------------------------------------------------------------------- 2022-03-17T21:19:05.7908814Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:05.7909138Z 2022-03-17T21:19:05.7909475Z ---------------------------------------------------------------------- 2022-03-17T21:19:05.7909788Z Ran 1 test in 0.002s 2022-03-17T21:19:05.7909947Z 2022-03-17T21:19:05.7910032Z OK (skipped=1) 2022-03-17T21:19:05.7910167Z 2022-03-17T21:19:05.7910309Z Generating XML reports... 2022-03-17T21:19:05.7955395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211905.xml 2022-03-17T21:19:06.9346678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:06.9360734Z 2022-03-17T21:19:06.9360942Z Running tests... 2022-03-17T21:19:06.9361353Z ---------------------------------------------------------------------- 2022-03-17T21:19:06.9380901Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:06.9381192Z 2022-03-17T21:19:06.9381477Z ---------------------------------------------------------------------- 2022-03-17T21:19:06.9381783Z Ran 1 test in 0.002s 2022-03-17T21:19:06.9381944Z 2022-03-17T21:19:06.9382060Z OK (skipped=1) 2022-03-17T21:19:06.9382216Z 2022-03-17T21:19:06.9382324Z Generating XML reports... 2022-03-17T21:19:06.9427294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211906.xml 2022-03-17T21:19:08.0829064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:08.0842902Z 2022-03-17T21:19:08.0843148Z Running tests... 2022-03-17T21:19:08.0844127Z ---------------------------------------------------------------------- 2022-03-17T21:19:08.0865188Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:08.0865494Z 2022-03-17T21:19:08.0865831Z ---------------------------------------------------------------------- 2022-03-17T21:19:08.0866377Z Ran 1 test in 0.002s 2022-03-17T21:19:08.0866498Z 2022-03-17T21:19:08.0866617Z OK (skipped=1) 2022-03-17T21:19:08.0866771Z 2022-03-17T21:19:08.0866895Z Generating XML reports... 2022-03-17T21:19:08.0911554Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211908.xml 2022-03-17T21:19:09.2322728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:09.2335800Z 2022-03-17T21:19:09.2336064Z Running tests... 2022-03-17T21:19:09.2336978Z ---------------------------------------------------------------------- 2022-03-17T21:19:10.7833274Z test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:10.8262591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58414 2022-03-17T21:19:10.8389858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58415 2022-03-17T21:19:11.9420894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:11.9421903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:11.9422687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:11.9423384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:11.9529005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:12.0435348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:14.2419932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:19:14.2421354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:19:14.2422138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:19:14.2422827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:19:14.6481189Z ok (5.414s) 2022-03-17T21:19:14.6481417Z 2022-03-17T21:19:14.6482034Z ---------------------------------------------------------------------- 2022-03-17T21:19:14.6482396Z Ran 1 test in 5.414s 2022-03-17T21:19:14.6482562Z 2022-03-17T21:19:14.6482635Z OK 2022-03-17T21:19:14.6482779Z 2022-03-17T21:19:14.6482927Z Generating XML reports... 2022-03-17T21:19:14.6526350Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211909.xml 2022-03-17T21:19:15.9532770Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:15.9547535Z 2022-03-17T21:19:15.9547669Z Running tests... 2022-03-17T21:19:15.9548618Z ---------------------------------------------------------------------- 2022-03-17T21:19:17.5073198Z test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:17.5516787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58543 2022-03-17T21:19:17.5648480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58544 2022-03-17T21:19:18.6564071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:18.6565087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:18.6566453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:18.6567134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:18.6673279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:18.7578624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:18.8695031Z skip (2.914s) 2022-03-17T21:19:18.8695451Z 2022-03-17T21:19:18.8696081Z ---------------------------------------------------------------------- 2022-03-17T21:19:18.8696425Z Ran 1 test in 2.915s 2022-03-17T21:19:18.8696557Z 2022-03-17T21:19:18.8696683Z OK (skipped=1) 2022-03-17T21:19:18.8696837Z 2022-03-17T21:19:18.8696965Z Generating XML reports... 2022-03-17T21:19:18.8740125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211915.xml 2022-03-17T21:19:20.1612423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:20.1626483Z 2022-03-17T21:19:20.1626749Z Running tests... 2022-03-17T21:19:20.1627176Z ---------------------------------------------------------------------- 2022-03-17T21:19:20.1647207Z test_backend_group (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:19:20.1647478Z 2022-03-17T21:19:20.1647755Z ---------------------------------------------------------------------- 2022-03-17T21:19:20.1648067Z Ran 1 test in 0.002s 2022-03-17T21:19:20.1648232Z 2022-03-17T21:19:20.1648371Z OK (skipped=1) 2022-03-17T21:19:20.1648440Z 2022-03-17T21:19:20.1648569Z Generating XML reports... 2022-03-17T21:19:20.1693171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211920.xml 2022-03-17T21:19:21.3110996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:21.3125128Z 2022-03-17T21:19:21.3125431Z Running tests... 2022-03-17T21:19:21.3125887Z ---------------------------------------------------------------------- 2022-03-17T21:19:22.8678145Z test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:22.9107838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58693 2022-03-17T21:19:22.9240026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58694 2022-03-17T21:19:24.0564367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:24.0565140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:24.0566136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:24.0566870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:24.0573084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:24.0573752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:25.0298337Z ok (3.717s) 2022-03-17T21:19:25.0298569Z 2022-03-17T21:19:25.0298989Z ---------------------------------------------------------------------- 2022-03-17T21:19:25.0299511Z Ran 1 test in 3.717s 2022-03-17T21:19:25.0299678Z 2022-03-17T21:19:25.0299749Z OK 2022-03-17T21:19:25.0299884Z 2022-03-17T21:19:25.0300030Z Generating XML reports... 2022-03-17T21:19:25.0342812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211921.xml 2022-03-17T21:19:26.3366647Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:26.3380830Z 2022-03-17T21:19:26.3381099Z Running tests... 2022-03-17T21:19:26.3381674Z ---------------------------------------------------------------------- 2022-03-17T21:19:27.8914144Z test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:27.9340059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58808 2022-03-17T21:19:27.9468857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58809 2022-03-17T21:19:29.0324466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:29.0325225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:29.0326035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:29.0326707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:29.0434033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:29.1338883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:31.5556001Z ok (5.217s) 2022-03-17T21:19:31.5556286Z 2022-03-17T21:19:31.5556606Z ---------------------------------------------------------------------- 2022-03-17T21:19:31.5556922Z Ran 1 test in 5.217s 2022-03-17T21:19:31.5557080Z 2022-03-17T21:19:31.5557162Z OK 2022-03-17T21:19:31.5557293Z 2022-03-17T21:19:31.5557427Z Generating XML reports... 2022-03-17T21:19:31.5598702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211926.xml 2022-03-17T21:19:32.8633569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:32.8648287Z 2022-03-17T21:19:32.8648628Z Running tests... 2022-03-17T21:19:32.8649044Z ---------------------------------------------------------------------- 2022-03-17T21:19:34.4064017Z test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:34.4495104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58925 2022-03-17T21:19:34.4625241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58926 2022-03-17T21:19:35.5772718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:35.5773235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:35.5774030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:35.5774710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:35.5781383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:35.5781857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:35.5991821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:19:35.5992393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:19:35.5993009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:19:35.5993687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:19:36.6691888Z ok (3.804s) 2022-03-17T21:19:36.6692185Z 2022-03-17T21:19:36.6692621Z ---------------------------------------------------------------------- 2022-03-17T21:19:36.6692961Z Ran 1 test in 3.804s 2022-03-17T21:19:36.6693347Z 2022-03-17T21:19:36.6693445Z OK 2022-03-17T21:19:36.6693584Z 2022-03-17T21:19:36.6693724Z Generating XML reports... 2022-03-17T21:19:36.6737451Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211932.xml 2022-03-17T21:19:37.9710767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:37.9724872Z 2022-03-17T21:19:37.9725242Z Running tests... 2022-03-17T21:19:37.9725732Z ---------------------------------------------------------------------- 2022-03-17T21:19:39.5188001Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:39.5619054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59046 2022-03-17T21:19:39.5747057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59047 2022-03-17T21:19:40.6790637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:40.6791192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:40.6791879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:40.6792571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:40.6901655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:40.7805916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:40.9794461Z skip (3.007s) 2022-03-17T21:19:40.9794669Z 2022-03-17T21:19:40.9795018Z ---------------------------------------------------------------------- 2022-03-17T21:19:40.9795639Z Ran 1 test in 3.007s 2022-03-17T21:19:40.9795825Z 2022-03-17T21:19:40.9795893Z OK (skipped=1) 2022-03-17T21:19:40.9796010Z 2022-03-17T21:19:40.9796136Z Generating XML reports... 2022-03-17T21:19:40.9838052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211937.xml 2022-03-17T21:19:42.2638797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:42.2652487Z 2022-03-17T21:19:42.2652752Z Running tests... 2022-03-17T21:19:42.2653195Z ---------------------------------------------------------------------- 2022-03-17T21:19:43.7939506Z test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:43.8366184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59161 2022-03-17T21:19:43.8493435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59162 2022-03-17T21:19:44.9364581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:44.9365532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:44.9366970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:44.9368294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:44.9373402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:44.9374283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:45.1540264Z skip (2.888s) 2022-03-17T21:19:45.1540614Z 2022-03-17T21:19:45.1541511Z ---------------------------------------------------------------------- 2022-03-17T21:19:45.1541902Z Ran 1 test in 2.889s 2022-03-17T21:19:45.1542072Z 2022-03-17T21:19:45.1542199Z OK (skipped=1) 2022-03-17T21:19:45.1542358Z 2022-03-17T21:19:45.1542724Z Generating XML reports... 2022-03-17T21:19:45.1584991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211942.xml 2022-03-17T21:19:46.4331253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:46.4346209Z 2022-03-17T21:19:46.4346511Z Running tests... 2022-03-17T21:19:46.4346960Z ---------------------------------------------------------------------- 2022-03-17T21:19:47.9564907Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:47.9990851Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59276 2022-03-17T21:19:48.0118730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59277 2022-03-17T21:19:49.1348203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:49.1348750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:49.1349478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:49.1350173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:49.1456256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:49.2361982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:49.4166116Z skip (2.982s) 2022-03-17T21:19:49.4166338Z 2022-03-17T21:19:49.4167259Z ---------------------------------------------------------------------- 2022-03-17T21:19:49.4167658Z Ran 1 test in 2.982s 2022-03-17T21:19:49.4167802Z 2022-03-17T21:19:49.4167890Z OK (skipped=1) 2022-03-17T21:19:49.4168033Z 2022-03-17T21:19:49.4168161Z Generating XML reports... 2022-03-17T21:19:49.4211580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211946.xml 2022-03-17T21:19:50.6982330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:50.6996450Z 2022-03-17T21:19:50.6996591Z Running tests... 2022-03-17T21:19:50.6997262Z ---------------------------------------------------------------------- 2022-03-17T21:19:52.2365235Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:52.2792137Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59391 2022-03-17T21:19:52.2921312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59392 2022-03-17T21:19:53.3900183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:53.3901051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:53.3901919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:53.3902577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:53.4008300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:53.4911047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:53.5118548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:19:53.5119333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:19:53.5120008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:19:53.5120697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:19:54.7984950Z ok (4.098s) 2022-03-17T21:19:54.7985150Z 2022-03-17T21:19:54.7985579Z ---------------------------------------------------------------------- 2022-03-17T21:19:54.7985961Z Ran 1 test in 4.099s 2022-03-17T21:19:54.7986077Z 2022-03-17T21:19:54.7986182Z OK 2022-03-17T21:19:54.7986297Z 2022-03-17T21:19:54.7986434Z Generating XML reports... 2022-03-17T21:19:54.8028718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211950.xml 2022-03-17T21:19:56.1042698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:19:56.1056814Z 2022-03-17T21:19:56.1057076Z Running tests... 2022-03-17T21:19:56.1057545Z ---------------------------------------------------------------------- 2022-03-17T21:19:57.6508103Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:19:57.6939983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59512 2022-03-17T21:19:57.7067769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59513 2022-03-17T21:19:58.8125589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:58.8126112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:58.8126910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:58.8127633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:58.8234202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:19:58.9137452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:19:58.9553891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:19:58.9554410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:19:58.9555100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:19:58.9555791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:00.2132106Z ok (4.107s) 2022-03-17T21:20:00.2132306Z 2022-03-17T21:20:00.2132716Z ---------------------------------------------------------------------- 2022-03-17T21:20:00.2133050Z Ran 1 test in 4.107s 2022-03-17T21:20:00.2133217Z 2022-03-17T21:20:00.2133308Z OK 2022-03-17T21:20:00.2133446Z 2022-03-17T21:20:00.2133561Z Generating XML reports... 2022-03-17T21:20:00.2175547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211956.xml 2022-03-17T21:20:01.5259384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:01.5274353Z 2022-03-17T21:20:01.5274617Z Running tests... 2022-03-17T21:20:01.5275050Z ---------------------------------------------------------------------- 2022-03-17T21:20:03.0888269Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:03.1333570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59633 2022-03-17T21:20:03.1471744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59634 2022-03-17T21:20:04.2460387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:04.2461230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:04.2462256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:04.2462936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:04.2469534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:04.2470016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:04.4517260Z skip (2.924s) 2022-03-17T21:20:04.4517443Z 2022-03-17T21:20:04.4517822Z ---------------------------------------------------------------------- 2022-03-17T21:20:04.4518346Z Ran 1 test in 2.924s 2022-03-17T21:20:04.4518505Z 2022-03-17T21:20:04.4518621Z OK (skipped=1) 2022-03-17T21:20:04.4518777Z 2022-03-17T21:20:04.4518906Z Generating XML reports... 2022-03-17T21:20:04.4561328Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212001.xml 2022-03-17T21:20:05.7376526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:05.7391958Z 2022-03-17T21:20:05.7392454Z Running tests... 2022-03-17T21:20:05.7392948Z ---------------------------------------------------------------------- 2022-03-17T21:20:07.2961254Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:07.3399591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59748 2022-03-17T21:20:07.3534872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59749 2022-03-17T21:20:08.4455575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:08.4456134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:08.4456907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:08.4457610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:08.4464068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:08.4464534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:08.6581374Z ok (2.919s) 2022-03-17T21:20:08.6581593Z 2022-03-17T21:20:08.6581967Z ---------------------------------------------------------------------- 2022-03-17T21:20:08.6582311Z Ran 1 test in 2.919s 2022-03-17T21:20:08.6582477Z 2022-03-17T21:20:08.6582587Z OK 2022-03-17T21:20:08.6583869Z 2022-03-17T21:20:08.6584252Z Generating XML reports... 2022-03-17T21:20:08.6625588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212005.xml 2022-03-17T21:20:09.9550477Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:09.9565747Z 2022-03-17T21:20:09.9565980Z Running tests... 2022-03-17T21:20:09.9566482Z ---------------------------------------------------------------------- 2022-03-17T21:20:11.5117954Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:11.5566783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59863 2022-03-17T21:20:11.5700845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59864 2022-03-17T21:20:12.6680883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:12.6681458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:12.6682215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:12.6683177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:12.6789445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:12.7692720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:12.9749919Z ok (3.018s) 2022-03-17T21:20:12.9750225Z 2022-03-17T21:20:12.9750779Z ---------------------------------------------------------------------- 2022-03-17T21:20:12.9751104Z Ran 1 test in 3.018s 2022-03-17T21:20:12.9751270Z 2022-03-17T21:20:12.9751363Z OK 2022-03-17T21:20:12.9753512Z 2022-03-17T21:20:12.9754112Z Generating XML reports... 2022-03-17T21:20:12.9794262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212009.xml 2022-03-17T21:20:14.2772782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:14.2788156Z 2022-03-17T21:20:14.2788584Z Running tests... 2022-03-17T21:20:14.2789076Z ---------------------------------------------------------------------- 2022-03-17T21:20:14.2822473Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:20:14.2822817Z 2022-03-17T21:20:14.2823173Z ---------------------------------------------------------------------- 2022-03-17T21:20:14.2823544Z Ran 1 test in 0.003s 2022-03-17T21:20:14.2823705Z 2022-03-17T21:20:14.2823825Z OK (skipped=1) 2022-03-17T21:20:14.2823963Z 2022-03-17T21:20:14.2824080Z Generating XML reports... 2022-03-17T21:20:14.2876245Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212014.xml 2022-03-17T21:20:15.4259501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:15.4274325Z 2022-03-17T21:20:15.4274645Z Running tests... 2022-03-17T21:20:15.4275116Z ---------------------------------------------------------------------- 2022-03-17T21:20:15.4304654Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:20:15.4304933Z 2022-03-17T21:20:15.4305238Z ---------------------------------------------------------------------- 2022-03-17T21:20:15.4305562Z Ran 1 test in 0.003s 2022-03-17T21:20:15.4305718Z 2022-03-17T21:20:15.4305816Z OK (skipped=1) 2022-03-17T21:20:15.4305972Z 2022-03-17T21:20:15.4306100Z Generating XML reports... 2022-03-17T21:20:15.4350817Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212015.xml 2022-03-17T21:20:16.5786380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:16.5801069Z 2022-03-17T21:20:16.5801299Z Running tests... 2022-03-17T21:20:16.5801746Z ---------------------------------------------------------------------- 2022-03-17T21:20:16.5838046Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.004s) 2022-03-17T21:20:16.5838341Z 2022-03-17T21:20:16.5838602Z ---------------------------------------------------------------------- 2022-03-17T21:20:16.5838940Z Ran 1 test in 0.004s 2022-03-17T21:20:16.5839115Z 2022-03-17T21:20:16.5839216Z OK (skipped=1) 2022-03-17T21:20:16.5839369Z 2022-03-17T21:20:16.5839509Z Generating XML reports... 2022-03-17T21:20:16.5890824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212016.xml 2022-03-17T21:20:17.7368019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:17.7383494Z 2022-03-17T21:20:17.7383715Z Running tests... 2022-03-17T21:20:17.7384177Z ---------------------------------------------------------------------- 2022-03-17T21:20:17.7413599Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:20:17.7414195Z 2022-03-17T21:20:17.7414520Z ---------------------------------------------------------------------- 2022-03-17T21:20:17.7414820Z Ran 1 test in 0.003s 2022-03-17T21:20:17.7414990Z 2022-03-17T21:20:17.7415082Z OK (skipped=1) 2022-03-17T21:20:17.7415186Z 2022-03-17T21:20:17.7415316Z Generating XML reports... 2022-03-17T21:20:17.7467881Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212017.xml 2022-03-17T21:20:18.9113455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:18.9128315Z 2022-03-17T21:20:18.9128701Z Running tests... 2022-03-17T21:20:18.9129456Z ---------------------------------------------------------------------- 2022-03-17T21:20:18.9155465Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:20:18.9155742Z 2022-03-17T21:20:18.9156191Z ---------------------------------------------------------------------- 2022-03-17T21:20:18.9156553Z Ran 1 test in 0.003s 2022-03-17T21:20:18.9156715Z 2022-03-17T21:20:18.9156829Z OK (skipped=1) 2022-03-17T21:20:18.9156972Z 2022-03-17T21:20:18.9157105Z Generating XML reports... 2022-03-17T21:20:18.9210189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212018.xml 2022-03-17T21:20:20.0549201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:20.0562784Z 2022-03-17T21:20:20.0562960Z Running tests... 2022-03-17T21:20:20.0563434Z ---------------------------------------------------------------------- 2022-03-17T21:20:20.0591253Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:20:20.0591550Z 2022-03-17T21:20:20.0591889Z ---------------------------------------------------------------------- 2022-03-17T21:20:20.0592209Z Ran 1 test in 0.003s 2022-03-17T21:20:20.0592372Z 2022-03-17T21:20:20.0592475Z OK (skipped=1) 2022-03-17T21:20:20.0592577Z 2022-03-17T21:20:20.0592684Z Generating XML reports... 2022-03-17T21:20:20.0636132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212020.xml 2022-03-17T21:20:21.2106167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:21.2124848Z 2022-03-17T21:20:21.2125051Z Running tests... 2022-03-17T21:20:21.2125476Z ---------------------------------------------------------------------- 2022-03-17T21:20:21.2153545Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:20:21.2153821Z 2022-03-17T21:20:21.2154133Z ---------------------------------------------------------------------- 2022-03-17T21:20:21.2154463Z Ran 1 test in 0.003s 2022-03-17T21:20:21.2154629Z 2022-03-17T21:20:21.2154744Z OK (skipped=1) 2022-03-17T21:20:21.2154900Z 2022-03-17T21:20:21.2155008Z Generating XML reports... 2022-03-17T21:20:21.2203909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212021.xml 2022-03-17T21:20:22.3608552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:22.3622249Z 2022-03-17T21:20:22.3622659Z Running tests... 2022-03-17T21:20:22.3623129Z ---------------------------------------------------------------------- 2022-03-17T21:20:23.8892861Z test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:23.9333630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60223 2022-03-17T21:20:23.9462910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60224 2022-03-17T21:20:25.0878348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:25.0878922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:25.0879951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:25.0880651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:25.0887243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:25.0887885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:25.3510117Z ok (2.988s) 2022-03-17T21:20:25.3510333Z 2022-03-17T21:20:25.3510712Z ---------------------------------------------------------------------- 2022-03-17T21:20:25.3511306Z Ran 1 test in 2.989s 2022-03-17T21:20:25.3511488Z 2022-03-17T21:20:25.3511584Z OK 2022-03-17T21:20:25.3511701Z 2022-03-17T21:20:25.3511838Z Generating XML reports... 2022-03-17T21:20:25.3552907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212022.xml 2022-03-17T21:20:26.6367981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:26.6386002Z 2022-03-17T21:20:26.6386327Z Running tests... 2022-03-17T21:20:26.6386867Z ---------------------------------------------------------------------- 2022-03-17T21:20:28.1927735Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:28.2364004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60342 2022-03-17T21:20:28.2503590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60343 2022-03-17T21:20:29.3294279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:29.3294832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:29.3295600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:29.3296304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:29.3402211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:29.4308331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:31.3582404Z ok (4.719s) 2022-03-17T21:20:31.3582648Z 2022-03-17T21:20:31.3583015Z ---------------------------------------------------------------------- 2022-03-17T21:20:31.3583337Z Ran 1 test in 4.720s 2022-03-17T21:20:31.3583471Z 2022-03-17T21:20:31.3583552Z OK 2022-03-17T21:20:31.3583689Z 2022-03-17T21:20:31.3583803Z Generating XML reports... 2022-03-17T21:20:31.3625616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212026.xml 2022-03-17T21:20:32.6633944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:32.6647934Z 2022-03-17T21:20:32.6648608Z Running tests... 2022-03-17T21:20:32.6649102Z ---------------------------------------------------------------------- 2022-03-17T21:20:34.1989205Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:34.2415376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60463 2022-03-17T21:20:34.2542519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60464 2022-03-17T21:20:35.3607673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:35.3608201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:35.3608972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:35.3609910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:35.3717471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:35.4618640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:35.4733264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:20:35.4733749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:20:35.4734614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:20:35.4735474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:20:35.7599516Z ok (3.095s) 2022-03-17T21:20:35.7599761Z 2022-03-17T21:20:35.7600150Z ---------------------------------------------------------------------- 2022-03-17T21:20:35.7600474Z Ran 1 test in 3.095s 2022-03-17T21:20:35.7600626Z 2022-03-17T21:20:35.7600746Z OK 2022-03-17T21:20:35.7600847Z 2022-03-17T21:20:35.7600981Z Generating XML reports... 2022-03-17T21:20:35.7646619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212032.xml 2022-03-17T21:20:37.0476343Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:37.0490122Z 2022-03-17T21:20:37.0490562Z Running tests... 2022-03-17T21:20:37.0491300Z ---------------------------------------------------------------------- 2022-03-17T21:20:38.5929671Z test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:38.6361022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60588 2022-03-17T21:20:38.6488659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60589 2022-03-17T21:20:39.7552447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:39.7552977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:39.7553747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:39.7554449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:39.7560674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:39.7561688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:39.9536501Z skip (2.904s) 2022-03-17T21:20:39.9536681Z 2022-03-17T21:20:39.9537080Z ---------------------------------------------------------------------- 2022-03-17T21:20:39.9537439Z Ran 1 test in 2.905s 2022-03-17T21:20:39.9537612Z 2022-03-17T21:20:39.9537705Z OK (skipped=1) 2022-03-17T21:20:39.9537863Z 2022-03-17T21:20:39.9537992Z Generating XML reports... 2022-03-17T21:20:39.9582514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212037.xml 2022-03-17T21:20:41.2441169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:41.2455630Z 2022-03-17T21:20:41.2456189Z Running tests... 2022-03-17T21:20:41.2456658Z ---------------------------------------------------------------------- 2022-03-17T21:20:42.7957137Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:42.8387610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60703 2022-03-17T21:20:42.8515965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60704 2022-03-17T21:20:43.9098599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:43.9099148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:43.9099929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:43.9100878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:43.9208490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:44.0112645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:45.5588541Z ok (4.313s) 2022-03-17T21:20:45.5588741Z 2022-03-17T21:20:45.5589158Z ---------------------------------------------------------------------- 2022-03-17T21:20:45.5589528Z Ran 1 test in 4.313s 2022-03-17T21:20:45.5589700Z 2022-03-17T21:20:45.5589772Z OK 2022-03-17T21:20:45.5589909Z 2022-03-17T21:20:45.5590052Z Generating XML reports... 2022-03-17T21:20:45.5632468Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212041.xml 2022-03-17T21:20:46.8586723Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:46.8600704Z 2022-03-17T21:20:46.8601028Z Running tests... 2022-03-17T21:20:46.8601714Z ---------------------------------------------------------------------- 2022-03-17T21:20:48.4022819Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:48.4453399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60820 2022-03-17T21:20:48.4584139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60821 2022-03-17T21:20:49.5622776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:49.5623355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:49.5624104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:49.5624785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:49.5631511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:49.5632282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:49.7630188Z ok (2.903s) 2022-03-17T21:20:49.7630382Z 2022-03-17T21:20:49.7630814Z ---------------------------------------------------------------------- 2022-03-17T21:20:49.7631112Z Ran 1 test in 2.903s 2022-03-17T21:20:49.7631282Z 2022-03-17T21:20:49.7631400Z OK 2022-03-17T21:20:49.7631519Z 2022-03-17T21:20:49.7631656Z Generating XML reports... 2022-03-17T21:20:49.7674472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212046.xml 2022-03-17T21:20:51.0487784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:51.0502292Z 2022-03-17T21:20:51.0502859Z Running tests... 2022-03-17T21:20:51.0503365Z ---------------------------------------------------------------------- 2022-03-17T21:20:52.6079008Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:52.6512693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60935 2022-03-17T21:20:52.6644169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60936 2022-03-17T21:20:53.7538243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:53.7539018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:53.7539816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:53.7540755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:53.7547831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:53.7548616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:53.7655433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:20:53.7656007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:20:53.7656652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:20:53.7657635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:20:53.7865016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:20:53.7865512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:20:53.7866138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:20:53.7866825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:20:55.4719659Z ok (4.421s) 2022-03-17T21:20:55.4719856Z 2022-03-17T21:20:55.4720291Z ---------------------------------------------------------------------- 2022-03-17T21:20:55.4720633Z Ran 1 test in 4.422s 2022-03-17T21:20:55.4720737Z 2022-03-17T21:20:55.4720831Z OK 2022-03-17T21:20:55.4720969Z 2022-03-17T21:20:55.4721105Z Generating XML reports... 2022-03-17T21:20:55.4766105Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212051.xml 2022-03-17T21:20:56.7802233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:20:56.7816991Z 2022-03-17T21:20:56.7817434Z Running tests... 2022-03-17T21:20:56.7817923Z ---------------------------------------------------------------------- 2022-03-17T21:20:58.3163574Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:20:58.3592700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61064 2022-03-17T21:20:58.3722366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61065 2022-03-17T21:20:59.4541187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:20:59.4541994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:20:59.4542845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:59.4543700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:20:59.4650047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:20:59.5553575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:20:59.5665786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:20:59.5666306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:20:59.5667608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:20:59.5668551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:20:59.5877115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:20:59.5877672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:20:59.5878801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:20:59.5879678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:21:01.2799557Z ok (4.498s) 2022-03-17T21:21:01.2799911Z 2022-03-17T21:21:01.2800327Z ---------------------------------------------------------------------- 2022-03-17T21:21:01.2800583Z Ran 1 test in 4.498s 2022-03-17T21:21:01.2800750Z 2022-03-17T21:21:01.2800842Z OK 2022-03-17T21:21:01.2801045Z 2022-03-17T21:21:01.2801188Z Generating XML reports... 2022-03-17T21:21:01.2843568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212056.xml 2022-03-17T21:21:02.5841125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:02.5855656Z 2022-03-17T21:21:02.5856268Z Running tests... 2022-03-17T21:21:02.5856895Z ---------------------------------------------------------------------- 2022-03-17T21:21:04.1291964Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:04.1718900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61193 2022-03-17T21:21:04.1845760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61194 2022-03-17T21:21:05.2412128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:05.2412942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:05.2413727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:05.2414656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:05.2520304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:05.3426683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:06.9196244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:06.9197055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:07.1924722Z ok (4.607s) 2022-03-17T21:21:07.1925030Z 2022-03-17T21:21:07.1925517Z ---------------------------------------------------------------------- 2022-03-17T21:21:07.1925859Z Ran 1 test in 4.607s 2022-03-17T21:21:07.1926028Z 2022-03-17T21:21:07.1926102Z OK 2022-03-17T21:21:07.1926236Z 2022-03-17T21:21:07.1926369Z Generating XML reports... 2022-03-17T21:21:07.1968648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212102.xml 2022-03-17T21:21:08.4914696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:08.4929047Z 2022-03-17T21:21:08.4929340Z Running tests... 2022-03-17T21:21:08.4929801Z ---------------------------------------------------------------------- 2022-03-17T21:21:10.0142282Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:10.0570918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61314 2022-03-17T21:21:10.0698680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61315 2022-03-17T21:21:11.1600796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:11.1601379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:11.1602071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:11.1602999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:11.1709710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:11.2615717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:12.8228653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:12.8229563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:12.8240690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:12.8241639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:13.1778188Z ok (4.685s) 2022-03-17T21:21:13.1778529Z 2022-03-17T21:21:13.1778972Z ---------------------------------------------------------------------- 2022-03-17T21:21:13.1779322Z Ran 1 test in 4.685s 2022-03-17T21:21:13.1779489Z 2022-03-17T21:21:13.1779582Z OK 2022-03-17T21:21:13.1779708Z 2022-03-17T21:21:13.1779821Z Generating XML reports... 2022-03-17T21:21:13.1823251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212108.xml 2022-03-17T21:21:14.4602405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:14.4617859Z 2022-03-17T21:21:14.4618267Z Running tests... 2022-03-17T21:21:14.4619046Z ---------------------------------------------------------------------- 2022-03-17T21:21:16.0037430Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:16.0458416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61435 2022-03-17T21:21:16.0584422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61436 2022-03-17T21:21:17.1684042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:17.1684531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:17.1685298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:17.1685996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:17.1693159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:17.1693667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:18.7941759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:18.7942285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:18.7957631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:18.7958131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:18.8114328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:18.8114794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:18.8128643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:18.8129094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:19.1662198Z ok (4.704s) 2022-03-17T21:21:19.1662358Z 2022-03-17T21:21:19.1663312Z ---------------------------------------------------------------------- 2022-03-17T21:21:19.1663844Z Ran 1 test in 4.704s 2022-03-17T21:21:19.1664115Z 2022-03-17T21:21:19.1664287Z OK 2022-03-17T21:21:19.1664537Z 2022-03-17T21:21:19.1664673Z Generating XML reports... 2022-03-17T21:21:19.1707176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212114.xml 2022-03-17T21:21:20.4596802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:20.4611121Z 2022-03-17T21:21:20.4611316Z Running tests... 2022-03-17T21:21:20.4611964Z ---------------------------------------------------------------------- 2022-03-17T21:21:21.9865160Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:22.0290725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61556 2022-03-17T21:21:22.0419440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61557 2022-03-17T21:21:23.1562117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:23.1562654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:23.1563435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:23.1564114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:23.1670713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:23.2576857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:24.8240964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:24.8241532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:24.8254126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:24.8254657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:24.8394353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:24.8394904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:24.8405748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:24.8406245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:25.1500688Z ok (4.689s) 2022-03-17T21:21:25.1501184Z 2022-03-17T21:21:25.1501581Z ---------------------------------------------------------------------- 2022-03-17T21:21:25.1501925Z Ran 1 test in 4.689s 2022-03-17T21:21:25.1502092Z 2022-03-17T21:21:25.1502173Z OK 2022-03-17T21:21:25.1502312Z 2022-03-17T21:21:25.1502447Z Generating XML reports... 2022-03-17T21:21:25.1544789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212120.xml 2022-03-17T21:21:26.4456895Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:26.4471364Z 2022-03-17T21:21:26.4471874Z Running tests... 2022-03-17T21:21:26.4472324Z ---------------------------------------------------------------------- 2022-03-17T21:21:27.9975155Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:28.0405529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61677 2022-03-17T21:21:28.0533458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61678 2022-03-17T21:21:29.1483553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:29.1484631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:29.1485401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:29.1486297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:29.1591719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:29.2497062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:30.5548205Z 2022-03-17T21:21:30.8616257Z ok (4.414s) 2022-03-17T21:21:30.8616603Z 2022-03-17T21:21:30.8617373Z ---------------------------------------------------------------------- 2022-03-17T21:21:30.8617792Z Ran 1 test in 4.414s 2022-03-17T21:21:30.8618056Z 2022-03-17T21:21:30.8618205Z OK 2022-03-17T21:21:30.8618496Z 2022-03-17T21:21:30.8618755Z Generating XML reports... 2022-03-17T21:21:30.8660841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212126.xml 2022-03-17T21:21:32.1616586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:32.1630647Z 2022-03-17T21:21:32.1631319Z Running tests... 2022-03-17T21:21:32.1632311Z ---------------------------------------------------------------------- 2022-03-17T21:21:33.6952152Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:33.7380812Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61794 2022-03-17T21:21:33.7507757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61795 2022-03-17T21:21:34.8385976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:34.8386766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:34.8387588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:34.8388250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:34.8395156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:34.8396199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:36.4585263Z ok (4.295s) 2022-03-17T21:21:36.4585584Z 2022-03-17T21:21:36.4586344Z ---------------------------------------------------------------------- 2022-03-17T21:21:36.4586874Z Ran 1 test in 4.295s 2022-03-17T21:21:36.4587056Z 2022-03-17T21:21:36.4587080Z OK 2022-03-17T21:21:36.4587246Z 2022-03-17T21:21:36.4587403Z Generating XML reports... 2022-03-17T21:21:36.4629109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212132.xml 2022-03-17T21:21:37.7654855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:37.7670372Z 2022-03-17T21:21:37.7670504Z Running tests... 2022-03-17T21:21:37.7671209Z ---------------------------------------------------------------------- 2022-03-17T21:21:39.3078244Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:39.3513996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61911 2022-03-17T21:21:39.3641973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61912 2022-03-17T21:21:40.4737234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:40.4737745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:40.4738598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:40.4739463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:40.4745666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:40.4747453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:42.1081014Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:42.1081608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:42.3729150Z ok (4.606s) 2022-03-17T21:21:42.3729362Z 2022-03-17T21:21:42.3729775Z ---------------------------------------------------------------------- 2022-03-17T21:21:42.3730103Z Ran 1 test in 4.606s 2022-03-17T21:21:42.3730251Z 2022-03-17T21:21:42.3730327Z OK 2022-03-17T21:21:42.3732969Z 2022-03-17T21:21:42.3733212Z Generating XML reports... 2022-03-17T21:21:42.3773075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212137.xml 2022-03-17T21:21:43.6238428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:43.6254011Z 2022-03-17T21:21:43.6254419Z Running tests... 2022-03-17T21:21:43.6254864Z ---------------------------------------------------------------------- 2022-03-17T21:21:45.1782151Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:45.2212893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62032 2022-03-17T21:21:45.2343161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62033 2022-03-17T21:21:46.3458860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:46.3459391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:46.3460172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:46.3471609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:46.3568831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:46.4474328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:48.0109305Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:21:48.3435957Z ok (4.718s) 2022-03-17T21:21:48.3436158Z 2022-03-17T21:21:48.3436549Z ---------------------------------------------------------------------- 2022-03-17T21:21:48.3436892Z Ran 1 test in 4.718s 2022-03-17T21:21:48.3437338Z 2022-03-17T21:21:48.3437430Z OK 2022-03-17T21:21:48.3437565Z 2022-03-17T21:21:48.3437679Z Generating XML reports... 2022-03-17T21:21:48.3479166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212143.xml 2022-03-17T21:21:49.6460504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:49.6475519Z 2022-03-17T21:21:49.6475787Z Running tests... 2022-03-17T21:21:49.6476230Z ---------------------------------------------------------------------- 2022-03-17T21:21:51.1878761Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:51.2312019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62153 2022-03-17T21:21:51.2440655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62154 2022-03-17T21:21:52.3429240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:52.3429808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:52.3430598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:52.3431297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:52.3537496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:52.4442999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:54.0156164Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:21:54.0157746Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:21:54.3519319Z ok (4.704s) 2022-03-17T21:21:54.3519551Z 2022-03-17T21:21:54.3519967Z ---------------------------------------------------------------------- 2022-03-17T21:21:54.3520305Z Ran 1 test in 4.704s 2022-03-17T21:21:54.3520469Z 2022-03-17T21:21:54.3520565Z OK 2022-03-17T21:21:54.3520705Z 2022-03-17T21:21:54.3520845Z Generating XML reports... 2022-03-17T21:21:54.3562388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212149.xml 2022-03-17T21:21:55.6616063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:55.6631284Z 2022-03-17T21:21:55.6631562Z Running tests... 2022-03-17T21:21:55.6632008Z ---------------------------------------------------------------------- 2022-03-17T21:21:57.1930108Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:21:57.2359800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62274 2022-03-17T21:21:57.2488633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62275 2022-03-17T21:21:58.3335706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:21:58.3336504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:21:58.3337292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:58.3337983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:21:58.3344749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:21:58.3345225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:21:58.3369944Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3371501Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3376725Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:994.) 2022-03-17T21:21:58.3377941Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T21:21:58.3379408Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:994.) 2022-03-17T21:21:58.3380658Z allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass 2022-03-17T21:21:58.3381217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:58.3381718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:21:58.3384180Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3385650Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3390206Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3391836Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3395405Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3396855Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3401178Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3402592Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3406203Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.3407639Z [W reducer.cpp:368] Using DistributedDataParallel with create_graph=True is not well-supported. The higher-order gradient will not be synchronized across ranks, and backpropagation through all_reduce operations will not occur. If you require DDP to work with higher-order gradients for your use case, please ping https://github.com/pytorch/pytorch/issues/63929 2022-03-17T21:21:58.5534530Z ok (2.890s) 2022-03-17T21:21:58.5534738Z 2022-03-17T21:21:58.5535175Z ---------------------------------------------------------------------- 2022-03-17T21:21:58.5535478Z Ran 1 test in 2.890s 2022-03-17T21:21:58.5535641Z 2022-03-17T21:21:58.5535738Z OK 2022-03-17T21:21:58.5535864Z 2022-03-17T21:21:58.5535990Z Generating XML reports... 2022-03-17T21:21:58.5577959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212155.xml 2022-03-17T21:21:59.8334501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:21:59.8349912Z 2022-03-17T21:21:59.8350331Z Running tests... 2022-03-17T21:21:59.8350839Z ---------------------------------------------------------------------- 2022-03-17T21:22:01.3959683Z test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:01.4418806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62393 2022-03-17T21:22:01.4563212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62394 2022-03-17T21:22:02.5791447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:02.5792217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:02.5793255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:02.5793952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:02.5900442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:02.6805264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:04.2599077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:04.2599589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:04.5643235Z ok (4.729s) 2022-03-17T21:22:04.5643447Z 2022-03-17T21:22:04.5644047Z ---------------------------------------------------------------------- 2022-03-17T21:22:04.5644503Z Ran 1 test in 4.729s 2022-03-17T21:22:04.5644682Z 2022-03-17T21:22:04.5644776Z OK 2022-03-17T21:22:04.5644913Z 2022-03-17T21:22:04.5645030Z Generating XML reports... 2022-03-17T21:22:04.5687704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212159.xml 2022-03-17T21:22:05.8726861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:05.8741305Z 2022-03-17T21:22:05.8741464Z Running tests... 2022-03-17T21:22:05.8742524Z ---------------------------------------------------------------------- 2022-03-17T21:22:07.4237676Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:07.4668590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62514 2022-03-17T21:22:07.4796661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62515 2022-03-17T21:22:08.5394836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:08.5395601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:08.5396560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:08.5397382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:08.5503435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:08.6408902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:09.9090831Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T21:22:09.9093175Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T21:22:09.9095607Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1033: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior. 2022-03-17T21:22:09.9097155Z warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes " 2022-03-17T21:22:10.4874589Z ok (4.613s) 2022-03-17T21:22:10.4875085Z 2022-03-17T21:22:10.4875473Z ---------------------------------------------------------------------- 2022-03-17T21:22:10.4875792Z Ran 1 test in 4.613s 2022-03-17T21:22:10.4875965Z 2022-03-17T21:22:10.4876059Z OK 2022-03-17T21:22:10.4876201Z 2022-03-17T21:22:10.4876336Z Generating XML reports... 2022-03-17T21:22:10.4917931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212205.xml 2022-03-17T21:22:11.7883278Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:11.7897381Z 2022-03-17T21:22:11.7897675Z Running tests... 2022-03-17T21:22:11.7898386Z ---------------------------------------------------------------------- 2022-03-17T21:22:13.3538654Z test_ddp_get_bucket_sizes (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:13.3970803Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62665 2022-03-17T21:22:13.4102053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62666 2022-03-17T21:22:14.4825166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:14.4825864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:14.4826651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:14.4827338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:14.4833547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:14.4834271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:19.4666459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:19.4667138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:22.4290475Z ok (10.639s) 2022-03-17T21:22:22.4290690Z 2022-03-17T21:22:22.4291161Z ---------------------------------------------------------------------- 2022-03-17T21:22:22.4291519Z Ran 1 test in 10.639s 2022-03-17T21:22:22.4291614Z 2022-03-17T21:22:22.4291688Z OK 2022-03-17T21:22:22.4293015Z 2022-03-17T21:22:22.4293271Z Generating XML reports... 2022-03-17T21:22:22.4337137Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212211.xml 2022-03-17T21:22:23.7305211Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:23.7319868Z 2022-03-17T21:22:23.7320266Z Running tests... 2022-03-17T21:22:23.7320755Z ---------------------------------------------------------------------- 2022-03-17T21:22:25.2629595Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:25.3057291Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62786 2022-03-17T21:22:25.3186187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62787 2022-03-17T21:22:26.4117384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:26.4117962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:26.4118703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:26.4119415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:26.4229486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:26.5131905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:28.0823772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:28.0824312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:28.4266935Z ok (4.694s) 2022-03-17T21:22:28.4267180Z 2022-03-17T21:22:28.4267589Z ---------------------------------------------------------------------- 2022-03-17T21:22:28.4267944Z Ran 1 test in 4.695s 2022-03-17T21:22:28.4268077Z 2022-03-17T21:22:28.4268170Z OK 2022-03-17T21:22:28.4268282Z 2022-03-17T21:22:28.4268422Z Generating XML reports... 2022-03-17T21:22:28.4312403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212223.xml 2022-03-17T21:22:29.7293907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:29.7308626Z 2022-03-17T21:22:29.7309163Z Running tests... 2022-03-17T21:22:29.7309695Z ---------------------------------------------------------------------- 2022-03-17T21:22:31.2600657Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:31.3030225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62907 2022-03-17T21:22:31.3159818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62908 2022-03-17T21:22:32.3935013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:32.3935571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:32.3936331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:32.3937027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:32.3942665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:32.3944034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:34.0081204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:34.0081712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:34.0097585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:34.0098061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:34.6242847Z ok (4.893s) 2022-03-17T21:22:34.6243067Z 2022-03-17T21:22:34.6243429Z ---------------------------------------------------------------------- 2022-03-17T21:22:34.6243781Z Ran 1 test in 4.893s 2022-03-17T21:22:34.6243953Z 2022-03-17T21:22:34.6244044Z OK 2022-03-17T21:22:34.6244159Z 2022-03-17T21:22:34.6244289Z Generating XML reports... 2022-03-17T21:22:34.6287030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212229.xml 2022-03-17T21:22:35.9210317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:35.9225284Z 2022-03-17T21:22:35.9225779Z Running tests... 2022-03-17T21:22:35.9226419Z ---------------------------------------------------------------------- 2022-03-17T21:22:37.4639339Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:37.5076264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63028 2022-03-17T21:22:37.5206446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63029 2022-03-17T21:22:38.6662470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:38.6663034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:38.6664265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:38.6665193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:38.6771561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:38.7672937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:38.7880633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:22:38.7881331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:22:38.7882190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:22:38.7882949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:22:40.4017913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:40.4018437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:40.4034730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:40.4035481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:41.0291046Z ok (5.106s) 2022-03-17T21:22:41.0291319Z 2022-03-17T21:22:41.0291730Z ---------------------------------------------------------------------- 2022-03-17T21:22:41.0292064Z Ran 1 test in 5.107s 2022-03-17T21:22:41.0292193Z 2022-03-17T21:22:41.0292265Z OK 2022-03-17T21:22:41.0292439Z 2022-03-17T21:22:41.0292531Z Generating XML reports... 2022-03-17T21:22:41.0335615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212235.xml 2022-03-17T21:22:42.3319564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:42.3334483Z 2022-03-17T21:22:42.3334936Z Running tests... 2022-03-17T21:22:42.3335463Z ---------------------------------------------------------------------- 2022-03-17T21:22:43.8766407Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:43.9200677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63155 2022-03-17T21:22:43.9328918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63156 2022-03-17T21:22:45.0402426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:45.0402988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:45.0403772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:45.0404467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:45.0513184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:45.0515475Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-03-17T21:22:45.1416793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:45.1417733Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations 2022-03-17T21:22:46.7227384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:46.7227951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:46.7243705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:46.7244186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:46.7490514Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-03-17T21:22:46.7491047Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations. 2022-03-17T21:22:47.0047630Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-03-17T21:22:47.0048230Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations 2022-03-17T21:22:47.0123309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:47.0123796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:47.0140955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:47.0141450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:47.6418279Z ok (5.308s) 2022-03-17T21:22:47.6418688Z 2022-03-17T21:22:47.6419093Z ---------------------------------------------------------------------- 2022-03-17T21:22:47.6419420Z Ran 1 test in 5.308s 2022-03-17T21:22:47.6419590Z 2022-03-17T21:22:47.6419748Z OK 2022-03-17T21:22:47.6419874Z 2022-03-17T21:22:47.6419941Z Generating XML reports... 2022-03-17T21:22:47.6461871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212242.xml 2022-03-17T21:22:48.9260848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:48.9274680Z 2022-03-17T21:22:48.9275040Z Running tests... 2022-03-17T21:22:48.9275508Z ---------------------------------------------------------------------- 2022-03-17T21:22:50.4507909Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:50.4929530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63276 2022-03-17T21:22:50.5055842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63277 2022-03-17T21:22:51.5750653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:51.5751222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:51.5751994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:51.5752669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:51.5861500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:51.5864128Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:22:51.6766781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:51.6768727Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:22:53.2363381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:53.2364176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:53.2380175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:53.2381194Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:53.2387106Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T21:22:53.2388114Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T21:22:53.2421686Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T21:22:53.2422294Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T21:22:53.2424439Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T21:22:53.2425065Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T21:22:53.2425688Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-03-17T21:22:53.2426452Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively. 2022-03-17T21:22:53.5650740Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:22:53.5651746Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 2; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:22:53.5728404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:53.5728936Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:53.5745488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:53.5745997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:53.5751343Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T21:22:53.5752381Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 2 iterations. 2022-03-17T21:22:53.5782115Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T21:22:53.5782789Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created. 2022-03-17T21:22:53.5783785Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T21:22:53.5784438Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 2, total before compression 10, total after compression 10, rate 1.0 2022-03-17T21:22:54.2146227Z ok (5.287s) 2022-03-17T21:22:54.2146527Z 2022-03-17T21:22:54.2146943Z ---------------------------------------------------------------------- 2022-03-17T21:22:54.2147279Z Ran 1 test in 5.287s 2022-03-17T21:22:54.2147431Z 2022-03-17T21:22:54.2147525Z OK 2022-03-17T21:22:54.2147635Z 2022-03-17T21:22:54.2148119Z Generating XML reports... 2022-03-17T21:22:54.2190706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212248.xml 2022-03-17T21:22:55.5012726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:22:55.5026968Z 2022-03-17T21:22:55.5027251Z Running tests... 2022-03-17T21:22:55.5027677Z ---------------------------------------------------------------------- 2022-03-17T21:22:57.0313677Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:22:57.0737464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63397 2022-03-17T21:22:57.0866465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63398 2022-03-17T21:22:58.1829635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:22:58.1830200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:22:58.1830977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:58.1831677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:22:58.1837457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:22:58.1837957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:22:59.5836781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplugstb5q 2022-03-17T21:22:59.5837342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw49x1fps 2022-03-17T21:22:59.5838123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplugstb5q/_remote_module_non_sriptable.py 2022-03-17T21:22:59.5838924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw49x1fps/_remote_module_non_sriptable.py 2022-03-17T21:22:59.9031069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:59.9031641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:59.9781662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:22:59.9782164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:00.3946211Z ok (4.892s) 2022-03-17T21:23:00.3946368Z 2022-03-17T21:23:00.3947472Z ---------------------------------------------------------------------- 2022-03-17T21:23:00.3948103Z Ran 1 test in 4.892s 2022-03-17T21:23:00.3948428Z 2022-03-17T21:23:00.3948570Z OK 2022-03-17T21:23:00.3948835Z 2022-03-17T21:23:00.3949043Z Generating XML reports... 2022-03-17T21:23:00.3992219Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212255.xml 2022-03-17T21:23:01.7106308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:01.7121202Z 2022-03-17T21:23:01.7121378Z Running tests... 2022-03-17T21:23:01.7121832Z ---------------------------------------------------------------------- 2022-03-17T21:23:03.2517730Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:03.2954119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63548 2022-03-17T21:23:03.3082602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63549 2022-03-17T21:23:04.3940055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:04.3940833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:04.3941900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:04.3942564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:04.3949577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:04.3950113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:05.7905943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4z811jow 2022-03-17T21:23:05.7907109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4z811jow/_remote_module_non_sriptable.py 2022-03-17T21:23:05.7908230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3o5j109 2022-03-17T21:23:05.7911191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3o5j109/_remote_module_non_sriptable.py 2022-03-17T21:23:06.1005133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:06.1005686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:06.1660837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:06.1661638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:06.5162777Z ok (4.804s) 2022-03-17T21:23:06.5162954Z 2022-03-17T21:23:06.5163546Z ---------------------------------------------------------------------- 2022-03-17T21:23:06.5164043Z Ran 1 test in 4.804s 2022-03-17T21:23:06.5164213Z 2022-03-17T21:23:06.5164293Z OK 2022-03-17T21:23:06.5164409Z 2022-03-17T21:23:06.5164544Z Generating XML reports... 2022-03-17T21:23:06.5207667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212301.xml 2022-03-17T21:23:07.8164821Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:07.8179353Z 2022-03-17T21:23:07.8179508Z Running tests... 2022-03-17T21:23:07.8180643Z ---------------------------------------------------------------------- 2022-03-17T21:23:09.3583606Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:09.4013717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63699 2022-03-17T21:23:09.4141739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63700 2022-03-17T21:23:10.5228958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:10.5229483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:10.5230317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:10.5230966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:10.5339050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:10.6243417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:11.9729686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgma8s9b 2022-03-17T21:23:11.9730573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6nk943e 2022-03-17T21:23:11.9731133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgma8s9b/_remote_module_non_sriptable.py 2022-03-17T21:23:11.9731652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6nk943e/_remote_module_non_sriptable.py 2022-03-17T21:23:12.2869183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:12.2869713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:12.3607359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:12.3607858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:12.7224194Z ok (4.904s) 2022-03-17T21:23:12.7224425Z 2022-03-17T21:23:12.7224809Z ---------------------------------------------------------------------- 2022-03-17T21:23:12.7225135Z Ran 1 test in 4.904s 2022-03-17T21:23:12.7225306Z 2022-03-17T21:23:12.7225606Z OK 2022-03-17T21:23:12.7225734Z 2022-03-17T21:23:12.7225867Z Generating XML reports... 2022-03-17T21:23:12.7269690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212307.xml 2022-03-17T21:23:14.0227511Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:14.0241881Z 2022-03-17T21:23:14.0242330Z Running tests... 2022-03-17T21:23:14.0242823Z ---------------------------------------------------------------------- 2022-03-17T21:23:15.5577425Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:15.6003012Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63850 2022-03-17T21:23:15.6129765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63851 2022-03-17T21:23:16.7048943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:16.7049475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:16.7050272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:16.7050979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:16.7158429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:16.8063324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:18.1442745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68q3ptjg 2022-03-17T21:23:18.1443376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45_mzml8 2022-03-17T21:23:18.1443964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68q3ptjg/_remote_module_non_sriptable.py 2022-03-17T21:23:18.1445031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45_mzml8/_remote_module_non_sriptable.py 2022-03-17T21:23:18.4588052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:18.4588624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:18.5323596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:18.5324110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:18.9210693Z ok (4.897s) 2022-03-17T21:23:18.9210883Z 2022-03-17T21:23:18.9211283Z ---------------------------------------------------------------------- 2022-03-17T21:23:18.9211592Z Ran 1 test in 4.897s 2022-03-17T21:23:18.9211755Z 2022-03-17T21:23:18.9211858Z OK 2022-03-17T21:23:18.9212020Z 2022-03-17T21:23:18.9212105Z Generating XML reports... 2022-03-17T21:23:18.9256459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212314.xml 2022-03-17T21:23:20.2083264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:20.2098276Z 2022-03-17T21:23:20.2098587Z Running tests... 2022-03-17T21:23:20.2099720Z ---------------------------------------------------------------------- 2022-03-17T21:23:21.7439586Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:21.7865511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64001 2022-03-17T21:23:21.7994736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64002 2022-03-17T21:23:22.8983273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:22.8983827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:22.8984625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:22.8985317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:22.9091658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:22.9997233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:24.3318422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1orjvslu 2022-03-17T21:23:24.3319001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_sh11g9 2022-03-17T21:23:24.3319568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1orjvslu/_remote_module_non_sriptable.py 2022-03-17T21:23:24.3320096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_sh11g9/_remote_module_non_sriptable.py 2022-03-17T21:23:24.6588282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:24.6588823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:24.7333205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:24.7333723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:25.1076604Z ok (4.897s) 2022-03-17T21:23:25.1076851Z 2022-03-17T21:23:25.1077253Z ---------------------------------------------------------------------- 2022-03-17T21:23:25.1077592Z Ran 1 test in 4.898s 2022-03-17T21:23:25.1077758Z 2022-03-17T21:23:25.1077840Z OK 2022-03-17T21:23:25.1077974Z 2022-03-17T21:23:25.1078121Z Generating XML reports... 2022-03-17T21:23:25.1120422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212320.xml 2022-03-17T21:23:26.4037224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:26.4051187Z 2022-03-17T21:23:26.4051463Z Running tests... 2022-03-17T21:23:26.4051921Z ---------------------------------------------------------------------- 2022-03-17T21:23:27.9479568Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:27.9908856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64152 2022-03-17T21:23:28.0038614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64153 2022-03-17T21:23:29.0649276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:29.0649803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:29.0650568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:29.0651521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:29.0758962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:29.1663441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:30.5095973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppv25zqbj 2022-03-17T21:23:30.5096580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewrqplm5 2022-03-17T21:23:30.5097351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppv25zqbj/_remote_module_non_sriptable.py 2022-03-17T21:23:30.5097893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewrqplm5/_remote_module_non_sriptable.py 2022-03-17T21:23:30.8323854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:30.8324400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:30.9001038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:30.9001566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:31.3122432Z ok (4.907s) 2022-03-17T21:23:31.3122648Z 2022-03-17T21:23:31.3123019Z ---------------------------------------------------------------------- 2022-03-17T21:23:31.3123371Z Ran 1 test in 4.907s 2022-03-17T21:23:31.3123541Z 2022-03-17T21:23:31.3123617Z OK 2022-03-17T21:23:31.3123752Z 2022-03-17T21:23:31.3123894Z Generating XML reports... 2022-03-17T21:23:31.3167555Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212326.xml 2022-03-17T21:23:32.6118670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:32.6132511Z 2022-03-17T21:23:32.6132779Z Running tests... 2022-03-17T21:23:32.6133231Z ---------------------------------------------------------------------- 2022-03-17T21:23:34.1622414Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:34.2047810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64303 2022-03-17T21:23:34.2177072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64304 2022-03-17T21:23:35.3265775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:35.3266353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:35.3267043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:35.3267745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:35.3374233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:35.4279481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:36.7943840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2r8eqane 2022-03-17T21:23:36.7944445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8q_z5wq 2022-03-17T21:23:36.7945531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8q_z5wq/_remote_module_non_sriptable.py 2022-03-17T21:23:36.7946091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2r8eqane/_remote_module_non_sriptable.py 2022-03-17T21:23:37.1068318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:37.1069344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:37.1812843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:37.1813784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:37.6261327Z ok (5.013s) 2022-03-17T21:23:37.6261614Z 2022-03-17T21:23:37.6262268Z ---------------------------------------------------------------------- 2022-03-17T21:23:37.6262770Z Ran 1 test in 5.013s 2022-03-17T21:23:37.6262925Z 2022-03-17T21:23:37.6262989Z OK 2022-03-17T21:23:37.6263125Z 2022-03-17T21:23:37.6263237Z Generating XML reports... 2022-03-17T21:23:37.6305927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212332.xml 2022-03-17T21:23:38.9338945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:38.9352926Z 2022-03-17T21:23:38.9353180Z Running tests... 2022-03-17T21:23:38.9353668Z ---------------------------------------------------------------------- 2022-03-17T21:23:40.4781591Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:40.5234101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64454 2022-03-17T21:23:40.5368369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64455 2022-03-17T21:23:41.6618201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:41.6618863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:41.6619661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:41.6620595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:41.6626295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:41.6627271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:43.0601748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gt7qt1c 2022-03-17T21:23:43.0602393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykmz8pa2 2022-03-17T21:23:43.0602925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gt7qt1c/_remote_module_non_sriptable.py 2022-03-17T21:23:43.0604107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykmz8pa2/_remote_module_non_sriptable.py 2022-03-17T21:23:43.3716734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:43.3717279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:43.4405348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:43.4405877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:43.8453646Z ok (4.910s) 2022-03-17T21:23:43.8453844Z 2022-03-17T21:23:43.8454224Z ---------------------------------------------------------------------- 2022-03-17T21:23:43.8454570Z Ran 1 test in 4.910s 2022-03-17T21:23:43.8454744Z 2022-03-17T21:23:43.8454839Z OK 2022-03-17T21:23:43.8454955Z 2022-03-17T21:23:43.8455088Z Generating XML reports... 2022-03-17T21:23:43.8498685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212338.xml 2022-03-17T21:23:45.1676011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:45.1691152Z 2022-03-17T21:23:45.1691595Z Running tests... 2022-03-17T21:23:45.1692044Z ---------------------------------------------------------------------- 2022-03-17T21:23:46.7089967Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:46.7521392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64605 2022-03-17T21:23:46.7646363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64606 2022-03-17T21:23:47.8938351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:47.8938913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:47.8939909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:47.8940959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:47.8947341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:47.8948403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:49.3144007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbaki6lww 2022-03-17T21:23:49.3144560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbswcjya 2022-03-17T21:23:49.3145116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbaki6lww/_remote_module_non_sriptable.py 2022-03-17T21:23:49.3145685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbswcjya/_remote_module_non_sriptable.py 2022-03-17T21:23:49.6413167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:49.6413687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:49.7099772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:49.7100529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:50.0728152Z ok (4.903s) 2022-03-17T21:23:50.0728395Z 2022-03-17T21:23:50.0728793Z ---------------------------------------------------------------------- 2022-03-17T21:23:50.0729131Z Ran 1 test in 4.904s 2022-03-17T21:23:50.0729279Z 2022-03-17T21:23:50.0729385Z OK 2022-03-17T21:23:50.0729517Z 2022-03-17T21:23:50.0729650Z Generating XML reports... 2022-03-17T21:23:50.0772427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212345.xml 2022-03-17T21:23:51.3421516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:51.3435924Z 2022-03-17T21:23:51.3436213Z Running tests... 2022-03-17T21:23:51.3436717Z ---------------------------------------------------------------------- 2022-03-17T21:23:52.8963395Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:52.9395342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64756 2022-03-17T21:23:52.9525403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64757 2022-03-17T21:23:54.0126844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:23:54.0127409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:23:54.0128227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:54.0128919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:23:54.0237012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:23:54.1140967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:23:55.4609479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7lx141v 2022-03-17T21:23:55.4610595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7lx141v/_remote_module_non_sriptable.py 2022-03-17T21:23:55.4613094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r7_twd7 2022-03-17T21:23:55.4616099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r7_twd7/_remote_module_non_sriptable.py 2022-03-17T21:23:55.7864782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:55.7865319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:55.8538122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:55.8538623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:23:56.2608759Z ok (4.917s) 2022-03-17T21:23:56.2609101Z 2022-03-17T21:23:56.2609556Z ---------------------------------------------------------------------- 2022-03-17T21:23:56.2609866Z Ran 1 test in 4.917s 2022-03-17T21:23:56.2610015Z 2022-03-17T21:23:56.2610160Z OK 2022-03-17T21:23:56.2610299Z 2022-03-17T21:23:56.2610428Z Generating XML reports... 2022-03-17T21:23:56.2654269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212351.xml 2022-03-17T21:23:57.5593303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:23:57.5607347Z 2022-03-17T21:23:57.5607764Z Running tests... 2022-03-17T21:23:57.5608262Z ---------------------------------------------------------------------- 2022-03-17T21:23:59.0852061Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:23:59.1279007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64907 2022-03-17T21:23:59.1407674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64908 2022-03-17T21:24:00.2061183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:00.2061742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:00.2062480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:00.2063172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:00.2070049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:00.2070543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:01.6178688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpmfcl5w 2022-03-17T21:24:01.6179275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn4r90kk 2022-03-17T21:24:01.6179789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpmfcl5w/_remote_module_non_sriptable.py 2022-03-17T21:24:01.6180533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn4r90kk/_remote_module_non_sriptable.py 2022-03-17T21:24:01.9301476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:01.9301944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:01.9984943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:01.9985501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:02.3488281Z ok (4.788s) 2022-03-17T21:24:02.3488612Z 2022-03-17T21:24:02.3489133Z ---------------------------------------------------------------------- 2022-03-17T21:24:02.3489473Z Ran 1 test in 4.788s 2022-03-17T21:24:02.3489639Z 2022-03-17T21:24:02.3489718Z OK 2022-03-17T21:24:02.3489858Z 2022-03-17T21:24:02.3489986Z Generating XML reports... 2022-03-17T21:24:02.3531682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212357.xml 2022-03-17T21:24:03.6495928Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:03.6511092Z 2022-03-17T21:24:03.6511329Z Running tests... 2022-03-17T21:24:03.6511861Z ---------------------------------------------------------------------- 2022-03-17T21:24:05.1986952Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:24:05.2416082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65058 2022-03-17T21:24:05.2544478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65059 2022-03-17T21:24:06.3150408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:06.3150929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:06.3151737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:06.3152433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:06.3158481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:06.3159195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:07.7072647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsyf3vwc 2022-03-17T21:24:07.7073381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmk3emj6e 2022-03-17T21:24:07.7074293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsyf3vwc/_remote_module_non_sriptable.py 2022-03-17T21:24:07.7075373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmk3emj6e/_remote_module_non_sriptable.py 2022-03-17T21:24:08.0156003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:08.0156573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:08.0831315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:08.4627890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:08.4628292Z ok (4.811s) 2022-03-17T21:24:08.4628428Z 2022-03-17T21:24:08.4628837Z ---------------------------------------------------------------------- 2022-03-17T21:24:08.4629178Z Ran 1 test in 4.812s 2022-03-17T21:24:08.4629343Z 2022-03-17T21:24:08.4629416Z OK 2022-03-17T21:24:08.4629549Z 2022-03-17T21:24:08.4629677Z Generating XML reports... 2022-03-17T21:24:08.4672903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212403.xml 2022-03-17T21:24:09.7615116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:09.7630003Z 2022-03-17T21:24:09.7630429Z Running tests... 2022-03-17T21:24:09.7630904Z ---------------------------------------------------------------------- 2022-03-17T21:24:11.2886085Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:24:11.3309710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65209 2022-03-17T21:24:11.3435296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65210 2022-03-17T21:24:12.4265103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:12.4265940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:12.4266740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:12.4267434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:12.4375693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:12.5279533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:14.0933332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.0934197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.1084891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.1085944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.1212926Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:24:14.1214515Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:24:14.1573984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.1575003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.1727453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.1728457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.1830725Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:24:14.1831583Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:24:14.1832731Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:24:14.1833544Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:24:14.1885543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.1886777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.2038146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.2039119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:14.5514425Z ok (4.788s) 2022-03-17T21:24:14.5514692Z 2022-03-17T21:24:14.5515408Z ---------------------------------------------------------------------- 2022-03-17T21:24:14.5516139Z Ran 1 test in 4.788s 2022-03-17T21:24:14.5516327Z 2022-03-17T21:24:14.5516412Z OK 2022-03-17T21:24:14.5516556Z 2022-03-17T21:24:14.5516883Z Generating XML reports... 2022-03-17T21:24:14.5559633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212409.xml 2022-03-17T21:24:15.8599705Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:15.8614196Z 2022-03-17T21:24:15.8614599Z Running tests... 2022-03-17T21:24:15.8615124Z ---------------------------------------------------------------------- 2022-03-17T21:24:17.4173605Z test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:24:17.4605114Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65330 2022-03-17T21:24:17.4734859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65331 2022-03-17T21:24:18.5537941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:18.5538526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:18.5539307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:18.5539993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:18.5546695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:18.5547130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:20.9823301Z ok (5.121s) 2022-03-17T21:24:20.9824157Z 2022-03-17T21:24:20.9824595Z ---------------------------------------------------------------------- 2022-03-17T21:24:20.9824953Z Ran 1 test in 5.121s 2022-03-17T21:24:20.9825098Z 2022-03-17T21:24:20.9825171Z OK 2022-03-17T21:24:20.9825312Z 2022-03-17T21:24:20.9825447Z Generating XML reports... 2022-03-17T21:24:20.9869911Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212415.xml 2022-03-17T21:24:22.2768374Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:22.2783232Z 2022-03-17T21:24:22.2783591Z Running tests... 2022-03-17T21:24:22.2784104Z ---------------------------------------------------------------------- 2022-03-17T21:24:23.8219017Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:24:23.8648169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65447 2022-03-17T21:24:23.8775934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65448 2022-03-17T21:24:24.9687280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:24.9687830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:24.9688641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:24.9689318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:24.9797315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:25.0701044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:26.6393721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:26.6394300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:26.9852948Z ok (4.707s) 2022-03-17T21:24:26.9853193Z 2022-03-17T21:24:26.9853574Z ---------------------------------------------------------------------- 2022-03-17T21:24:26.9853912Z Ran 1 test in 4.707s 2022-03-17T21:24:26.9854063Z 2022-03-17T21:24:26.9854423Z OK 2022-03-17T21:24:26.9854547Z 2022-03-17T21:24:26.9854691Z Generating XML reports... 2022-03-17T21:24:26.9896742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212422.xml 2022-03-17T21:24:28.2871648Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:28.2889394Z 2022-03-17T21:24:28.2890029Z Running tests... 2022-03-17T21:24:28.2890759Z ---------------------------------------------------------------------- 2022-03-17T21:24:29.8485159Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:24:29.8917636Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65568 2022-03-17T21:24:29.9048689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65569 2022-03-17T21:24:30.9949946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:30.9950462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:30.9951205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:30.9951900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:30.9958207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:30.9958700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:31.0014469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:31.0014955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:31.2096768Z ok (2.920s) 2022-03-17T21:24:31.2096996Z 2022-03-17T21:24:31.2097407Z ---------------------------------------------------------------------- 2022-03-17T21:24:31.2097726Z Ran 1 test in 2.921s 2022-03-17T21:24:31.2097888Z 2022-03-17T21:24:31.2097980Z OK 2022-03-17T21:24:31.2100395Z 2022-03-17T21:24:31.2100573Z Generating XML reports... 2022-03-17T21:24:31.2139357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212428.xml 2022-03-17T21:24:32.4863060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:32.4877263Z 2022-03-17T21:24:32.4877661Z Running tests... 2022-03-17T21:24:32.4878153Z ---------------------------------------------------------------------- 2022-03-17T21:24:34.0358340Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:24:34.0788883Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65717 2022-03-17T21:24:34.0918477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65718 2022-03-17T21:24:35.1923606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:35.1924599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:35.1925672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:35.1926361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:35.1931854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:35.1932293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:36.8276657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:36.8277474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:24:37.1998857Z ok (4.712s) 2022-03-17T21:24:37.1999083Z 2022-03-17T21:24:37.1999798Z ---------------------------------------------------------------------- 2022-03-17T21:24:37.2000426Z Ran 1 test in 4.712s 2022-03-17T21:24:37.2000728Z 2022-03-17T21:24:37.2000830Z OK 2022-03-17T21:24:37.2000966Z 2022-03-17T21:24:37.2001082Z Generating XML reports... 2022-03-17T21:24:37.2044181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212432.xml 2022-03-17T21:24:38.5068083Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:38.5081968Z 2022-03-17T21:24:38.5082522Z Running tests... 2022-03-17T21:24:38.5083000Z ---------------------------------------------------------------------- 2022-03-17T21:24:40.0488915Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:24:40.0915042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65838 2022-03-17T21:24:40.1044786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65839 2022-03-17T21:24:41.2168628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:41.2169573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:41.2171004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:41.2172367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:41.2177979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:41.2178864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:41.2388226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:24:41.2389210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:24:41.2390520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:24:41.2391926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:24:41.2496724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:24:41.2497615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:24:41.2498758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:24:41.2499933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:24:42.9120154Z ok (4.403s) 2022-03-17T21:24:42.9120415Z 2022-03-17T21:24:42.9120803Z ---------------------------------------------------------------------- 2022-03-17T21:24:42.9121435Z Ran 1 test in 4.404s 2022-03-17T21:24:42.9121586Z 2022-03-17T21:24:42.9121682Z OK 2022-03-17T21:24:42.9121820Z 2022-03-17T21:24:42.9124193Z Generating XML reports... 2022-03-17T21:24:42.9164647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212438.xml 2022-03-17T21:24:44.1882529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:44.1898214Z 2022-03-17T21:24:44.1898739Z Running tests... 2022-03-17T21:24:44.1899225Z ---------------------------------------------------------------------- 2022-03-17T21:24:45.7310834Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:24:45.7740834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65967 2022-03-17T21:24:45.7870637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65968 2022-03-17T21:24:46.8502278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:24:46.8502815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:24:46.8503521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:46.8504218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:24:46.8610279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:24:46.9512945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:24:46.9722045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:24:46.9722542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:24:46.9723181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:24:46.9723850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:24:46.9829584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:24:46.9830185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:24:46.9830766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:24:46.9831438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:24:58.6102530Z ok (14.420s) 2022-03-17T21:24:58.6102743Z 2022-03-17T21:24:58.6103101Z ---------------------------------------------------------------------- 2022-03-17T21:24:58.6103463Z Ran 1 test in 14.420s 2022-03-17T21:24:58.6103629Z 2022-03-17T21:24:58.6103728Z OK 2022-03-17T21:24:58.6103847Z 2022-03-17T21:24:58.6103981Z Generating XML reports... 2022-03-17T21:24:58.6146510Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212444.xml 2022-03-17T21:24:59.9155818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:24:59.9170209Z 2022-03-17T21:24:59.9170690Z Running tests... 2022-03-17T21:24:59.9171175Z ---------------------------------------------------------------------- 2022-03-17T21:25:01.4737652Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:01.5181009Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66096 2022-03-17T21:25:01.5317804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66097 2022-03-17T21:25:02.5939301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:02.5939944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:02.5941040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:02.5941697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:02.5947898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:02.5948907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:05.0405333Z ok (5.123s) 2022-03-17T21:25:05.0405546Z 2022-03-17T21:25:05.0405972Z ---------------------------------------------------------------------- 2022-03-17T21:25:05.0406329Z Ran 1 test in 5.123s 2022-03-17T21:25:05.0406496Z 2022-03-17T21:25:05.0406548Z OK 2022-03-17T21:25:05.0408684Z 2022-03-17T21:25:05.0409365Z Generating XML reports... 2022-03-17T21:25:05.0449818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212459.xml 2022-03-17T21:25:06.3466616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:06.3481459Z 2022-03-17T21:25:06.3481750Z Running tests... 2022-03-17T21:25:06.3482156Z ---------------------------------------------------------------------- 2022-03-17T21:25:07.8845209Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:07.9289953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66217 2022-03-17T21:25:07.9424981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66218 2022-03-17T21:25:09.0632358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:09.0632900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:09.0633621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:09.0634330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:09.0641411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:09.0641847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:11.5514059Z ok (5.203s) 2022-03-17T21:25:11.5514281Z 2022-03-17T21:25:11.5514682Z ---------------------------------------------------------------------- 2022-03-17T21:25:11.5515011Z Ran 1 test in 5.203s 2022-03-17T21:25:11.5515178Z 2022-03-17T21:25:11.5515224Z OK 2022-03-17T21:25:11.5515362Z 2022-03-17T21:25:11.5515494Z Generating XML reports... 2022-03-17T21:25:11.5557239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212506.xml 2022-03-17T21:25:12.8525205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:12.8539562Z 2022-03-17T21:25:12.8540022Z Running tests... 2022-03-17T21:25:12.8540804Z ---------------------------------------------------------------------- 2022-03-17T21:25:14.3964921Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:14.4392450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66338 2022-03-17T21:25:14.4519961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66339 2022-03-17T21:25:15.5712826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:15.5713612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:15.5714391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:15.5715059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:15.5721709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:15.5722182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:17.5607093Z ok (4.706s) 2022-03-17T21:25:17.5607396Z 2022-03-17T21:25:17.5607798Z ---------------------------------------------------------------------- 2022-03-17T21:25:17.5608111Z Ran 1 test in 4.707s 2022-03-17T21:25:17.5608258Z 2022-03-17T21:25:17.5608353Z OK 2022-03-17T21:25:17.5608509Z 2022-03-17T21:25:17.5608643Z Generating XML reports... 2022-03-17T21:25:17.5651396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212512.xml 2022-03-17T21:25:18.8569038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:18.8583297Z 2022-03-17T21:25:18.8583435Z Running tests... 2022-03-17T21:25:18.8584144Z ---------------------------------------------------------------------- 2022-03-17T21:25:20.4062595Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:20.4495528Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66455 2022-03-17T21:25:20.4623371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66456 2022-03-17T21:25:21.5372481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:21.5373052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:21.5373827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:21.5374535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:21.5481235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:21.6388799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:23.2354333Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:23.2360281Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:23.2444843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:23.2445810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:23.5703268Z ok (4.712s) 2022-03-17T21:25:23.5703816Z 2022-03-17T21:25:23.5704208Z ---------------------------------------------------------------------- 2022-03-17T21:25:23.5704523Z Ran 1 test in 4.712s 2022-03-17T21:25:23.5704669Z 2022-03-17T21:25:23.5704790Z OK 2022-03-17T21:25:23.5704906Z 2022-03-17T21:25:23.5705013Z Generating XML reports... 2022-03-17T21:25:23.5746714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212518.xml 2022-03-17T21:25:24.8683833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:24.8699229Z 2022-03-17T21:25:24.8699724Z Running tests... 2022-03-17T21:25:24.8701352Z ---------------------------------------------------------------------- 2022-03-17T21:25:26.4110607Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:26.4541718Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66576 2022-03-17T21:25:26.4669949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66577 2022-03-17T21:25:27.5283031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:27.5283571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:27.5284340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:27.5285016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:27.5291662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:27.5292125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:28.8704615Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:25:28.8705515Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:25:28.8706669Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:25:28.8707499Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:25:29.4747866Z ok (4.605s) 2022-03-17T21:25:29.4748095Z 2022-03-17T21:25:29.4748496Z ---------------------------------------------------------------------- 2022-03-17T21:25:29.4748834Z Ran 1 test in 4.605s 2022-03-17T21:25:29.4748982Z 2022-03-17T21:25:29.4749076Z OK 2022-03-17T21:25:29.4749212Z 2022-03-17T21:25:29.4749345Z Generating XML reports... 2022-03-17T21:25:29.4791069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212524.xml 2022-03-17T21:25:30.7763961Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:30.7778125Z 2022-03-17T21:25:30.7778533Z Running tests... 2022-03-17T21:25:30.7779040Z ---------------------------------------------------------------------- 2022-03-17T21:25:32.3300036Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:32.3735574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66697 2022-03-17T21:25:32.3871926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66698 2022-03-17T21:25:33.4766577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:33.4767118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:33.4767898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:33.4768576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:33.4775749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:33.4776241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:35.5080766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:35.5081315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:35.5400061Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:35.5401634Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:35.8974587Z ok (5.119s) 2022-03-17T21:25:35.8974819Z 2022-03-17T21:25:35.8975188Z ---------------------------------------------------------------------- 2022-03-17T21:25:35.8975540Z Ran 1 test in 5.120s 2022-03-17T21:25:35.8975667Z 2022-03-17T21:25:35.8975762Z OK 2022-03-17T21:25:35.8975878Z 2022-03-17T21:25:35.8976014Z Generating XML reports... 2022-03-17T21:25:35.9018999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212530.xml 2022-03-17T21:25:37.2014922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:37.2030003Z 2022-03-17T21:25:37.2030362Z Running tests... 2022-03-17T21:25:37.2030785Z ---------------------------------------------------------------------- 2022-03-17T21:25:38.7443435Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:38.7873348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66822 2022-03-17T21:25:38.8001824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66823 2022-03-17T21:25:39.8572697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:39.8575070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:39.8575865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:39.8675083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:39.8681861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:39.9591022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:41.9315527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:41.9316050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:41.9677236Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:41.9678813Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:42.3087710Z ok (5.105s) 2022-03-17T21:25:42.3087902Z 2022-03-17T21:25:42.3088296Z ---------------------------------------------------------------------- 2022-03-17T21:25:42.3088618Z Ran 1 test in 5.106s 2022-03-17T21:25:42.3088741Z 2022-03-17T21:25:42.3088835Z OK 2022-03-17T21:25:42.3088986Z 2022-03-17T21:25:42.3089100Z Generating XML reports... 2022-03-17T21:25:42.3130774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212537.xml 2022-03-17T21:25:43.6074378Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:43.6088617Z 2022-03-17T21:25:43.6088761Z Running tests... 2022-03-17T21:25:43.6089490Z ---------------------------------------------------------------------- 2022-03-17T21:25:45.1672363Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:45.2104510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66947 2022-03-17T21:25:45.2234773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66948 2022-03-17T21:25:46.3170898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:46.3171658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:46.3172456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:46.3173128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:46.3179099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:46.3179899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:47.9318391Z ok (4.323s) 2022-03-17T21:25:47.9318598Z 2022-03-17T21:25:47.9319000Z ---------------------------------------------------------------------- 2022-03-17T21:25:47.9319342Z Ran 1 test in 4.323s 2022-03-17T21:25:47.9319511Z 2022-03-17T21:25:47.9319605Z OK 2022-03-17T21:25:47.9319742Z 2022-03-17T21:25:47.9319859Z Generating XML reports... 2022-03-17T21:25:47.9362403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212543.xml 2022-03-17T21:25:49.2362711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:49.2376823Z 2022-03-17T21:25:49.2377122Z Running tests... 2022-03-17T21:25:49.2377547Z ---------------------------------------------------------------------- 2022-03-17T21:25:50.7792657Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:50.8214614Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67064 2022-03-17T21:25:50.8340600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67065 2022-03-17T21:25:51.9477666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:51.9478362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:51.9479169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:51.9479860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:51.9587377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:52.0491808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:53.3269091Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:25:53.3270001Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:25:53.3271160Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:25:53.3271977Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:25:53.6195757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:53.6196268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:53.6265901Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:53.6267449Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:25:53.6406431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:53.6406910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:53.6489664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:53.6490150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:53.9420364Z ok (4.704s) 2022-03-17T21:25:53.9420950Z 2022-03-17T21:25:53.9421426Z ---------------------------------------------------------------------- 2022-03-17T21:25:53.9421770Z Ran 1 test in 4.704s 2022-03-17T21:25:53.9421935Z 2022-03-17T21:25:53.9422026Z OK 2022-03-17T21:25:53.9422143Z 2022-03-17T21:25:53.9422279Z Generating XML reports... 2022-03-17T21:25:53.9464445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212549.xml 2022-03-17T21:25:55.2451226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:25:55.2466111Z 2022-03-17T21:25:55.2466275Z Running tests... 2022-03-17T21:25:55.2466719Z ---------------------------------------------------------------------- 2022-03-17T21:25:56.7881350Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:25:56.8310158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67185 2022-03-17T21:25:56.8438804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67186 2022-03-17T21:25:57.9318315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:25:57.9318847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:25:57.9319600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:57.9320306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:25:57.9429210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:25:58.0333076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:25:59.3158989Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:25:59.3159890Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:25:59.3161050Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:1770: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`. 2022-03-17T21:25:59.3161865Z "You passed find_unused_parameters=true to DistributedDataParallel, " 2022-03-17T21:25:59.6063685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:59.6064261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:25:59.9517857Z ok (4.705s) 2022-03-17T21:25:59.9518067Z 2022-03-17T21:25:59.9518517Z ---------------------------------------------------------------------- 2022-03-17T21:25:59.9518843Z Ran 1 test in 4.705s 2022-03-17T21:25:59.9519020Z 2022-03-17T21:25:59.9519109Z OK 2022-03-17T21:25:59.9519254Z 2022-03-17T21:25:59.9519304Z Generating XML reports... 2022-03-17T21:25:59.9563592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212555.xml 2022-03-17T21:26:01.2596077Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:01.2610870Z 2022-03-17T21:26:01.2611082Z Running tests... 2022-03-17T21:26:01.2611682Z ---------------------------------------------------------------------- 2022-03-17T21:26:02.8202158Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:26:02.8628813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67306 2022-03-17T21:26:02.8758918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67307 2022-03-17T21:26:03.9483636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:03.9484177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:03.9485225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:03.9485950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:03.9591618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:04.0498587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:05.6245780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:05.6246304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:05.6347809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:05.6348324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:05.9837507Z ok (4.722s) 2022-03-17T21:26:05.9837765Z 2022-03-17T21:26:05.9838198Z ---------------------------------------------------------------------- 2022-03-17T21:26:05.9838542Z Ran 1 test in 4.723s 2022-03-17T21:26:05.9838664Z 2022-03-17T21:26:05.9838754Z OK 2022-03-17T21:26:05.9838888Z 2022-03-17T21:26:05.9839019Z Generating XML reports... 2022-03-17T21:26:05.9881113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212601.xml 2022-03-17T21:26:07.2842045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:07.2857293Z 2022-03-17T21:26:07.2857405Z Running tests... 2022-03-17T21:26:07.2858186Z ---------------------------------------------------------------------- 2022-03-17T21:26:08.8114190Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:26:08.8547878Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67427 2022-03-17T21:26:08.8676017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67428 2022-03-17T21:26:09.9855818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:09.9856362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:09.9857142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:09.9857832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:09.9865052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:09.9865523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:11.6803556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:11.6804103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:12.5765869Z ok (5.290s) 2022-03-17T21:26:12.5766073Z 2022-03-17T21:26:12.5766489Z ---------------------------------------------------------------------- 2022-03-17T21:26:12.5767081Z Ran 1 test in 5.291s 2022-03-17T21:26:12.5767245Z 2022-03-17T21:26:12.5767334Z OK 2022-03-17T21:26:12.5767467Z 2022-03-17T21:26:12.5767602Z Generating XML reports... 2022-03-17T21:26:12.5810340Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212607.xml 2022-03-17T21:26:13.8366529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:13.8381105Z 2022-03-17T21:26:13.8381601Z Running tests... 2022-03-17T21:26:13.8382083Z ---------------------------------------------------------------------- 2022-03-17T21:26:15.4049328Z test_ddp_sync_params_and_buffers (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:26:15.4484791Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67552 2022-03-17T21:26:15.4614376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67553 2022-03-17T21:26:16.5160793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:16.5161729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:16.5162526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:16.5163200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:16.5272277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:16.6175858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:18.1696433Z ok (4.331s) 2022-03-17T21:26:18.1696838Z 2022-03-17T21:26:18.1697230Z ---------------------------------------------------------------------- 2022-03-17T21:26:18.1697568Z Ran 1 test in 4.332s 2022-03-17T21:26:18.1697716Z 2022-03-17T21:26:18.1697829Z OK 2022-03-17T21:26:18.1697968Z 2022-03-17T21:26:18.1698101Z Generating XML reports... 2022-03-17T21:26:18.1739587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212613.xml 2022-03-17T21:26:19.4722149Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:19.4736951Z 2022-03-17T21:26:19.4737334Z Running tests... 2022-03-17T21:26:19.4737898Z ---------------------------------------------------------------------- 2022-03-17T21:26:21.0201527Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:26:21.0646820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67669 2022-03-17T21:26:21.0780101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67670 2022-03-17T21:26:22.1995046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:22.1995823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:22.1996754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:22.1997598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:22.2103284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:22.3008871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:23.8855459Z ok (4.411s) 2022-03-17T21:26:23.8855672Z 2022-03-17T21:26:23.8856076Z ---------------------------------------------------------------------- 2022-03-17T21:26:23.8856420Z Ran 1 test in 4.412s 2022-03-17T21:26:23.8856592Z 2022-03-17T21:26:23.8856686Z OK 2022-03-17T21:26:23.8856825Z 2022-03-17T21:26:23.8857236Z Generating XML reports... 2022-03-17T21:26:23.8901100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212619.xml 2022-03-17T21:26:25.1836339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:25.1848720Z 2022-03-17T21:26:25.1849139Z Running tests... 2022-03-17T21:26:25.1849641Z ---------------------------------------------------------------------- 2022-03-17T21:26:26.7252523Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:26:26.7681514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67786 2022-03-17T21:26:26.7810668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67787 2022-03-17T21:26:27.8959936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:27.8960539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:27.8961335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:27.8962006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:27.9068897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:27.9974252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:29.5579905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:29.5580663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:29.8901102Z ok (4.705s) 2022-03-17T21:26:29.8901264Z 2022-03-17T21:26:29.8901855Z ---------------------------------------------------------------------- 2022-03-17T21:26:29.8902205Z Ran 1 test in 4.705s 2022-03-17T21:26:29.8902352Z 2022-03-17T21:26:29.8902422Z OK 2022-03-17T21:26:29.8902560Z 2022-03-17T21:26:29.8902686Z Generating XML reports... 2022-03-17T21:26:29.8945275Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212625.xml 2022-03-17T21:26:31.1876974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:26:31.1890791Z 2022-03-17T21:26:31.1891197Z Running tests... 2022-03-17T21:26:31.1891706Z ---------------------------------------------------------------------- 2022-03-17T21:26:32.7274797Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:26:32.7699838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67907 2022-03-17T21:26:32.7828531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67908 2022-03-17T21:26:33.8731779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:26:33.8732315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:26:33.8733062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:33.8733759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:26:33.8740868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:26:33.8741351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:26:35.2096268Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:26:35.2129264Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000 2022-03-17T21:26:35.2134430Z Running test: batch_norm_net sync interval 2022-03-17T21:26:35.2134741Z 1 with iteration mapping 2022-03-17T21:26:35.2134959Z {0: 0, 1: 2} 2022-03-17T21:26:35.8561015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:35.8609724Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:35.8610536Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:35.8611508Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:35.8612268Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:35.8716368Z Running test: batch_norm_net sync interval 2022-03-17T21:26:35.8716656Z 1 with iteration mapping 2022-03-17T21:26:35.8716903Z {0: 0, 1: 3} 2022-03-17T21:26:35.8777744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:35.8849364Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:35.8850157Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:35.8851173Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:35.8851935Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:35.8931924Z Running test: batch_norm_net sync interval 2022-03-17T21:26:35.8932197Z 1 with iteration mapping 2022-03-17T21:26:35.8932471Z {0: 0, 1: 10} 2022-03-17T21:26:35.8992296Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:35.9250661Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:35.9252086Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:35.9253164Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:35.9254116Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:35.9342261Z Running test: batch_norm_net sync interval 2022-03-17T21:26:35.9342562Z 1 with iteration mapping 2022-03-17T21:26:35.9342820Z {0: 5, 1: 7} 2022-03-17T21:26:36.5650417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:36.5650949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:36.5830226Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:36.5831025Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:36.5832064Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:36.5832805Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:36.5923114Z Running test: batch_norm_net sync interval 2022-03-17T21:26:36.5923384Z 1 with iteration mapping 2022-03-17T21:26:36.5923638Z {0: 5, 1: 8} 2022-03-17T21:26:36.5983254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:36.5983670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:36.6192178Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:36.6193003Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:36.6193942Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:36.6194696Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:36.6274590Z Running test: batch_norm_net sync interval 2022-03-17T21:26:36.6274859Z 1 with iteration mapping 2022-03-17T21:26:36.6275116Z {0: 5, 1: 15} 2022-03-17T21:26:36.6340229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:36.6340691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:36.6758010Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:36.6758712Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:36.6759724Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:36.6760654Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:36.6841942Z Running test: large_conv_model sync interval 2022-03-17T21:26:36.6842197Z 1 with iteration mapping 2022-03-17T21:26:36.6842463Z {0: 0, 1: 2} 2022-03-17T21:26:38.4974576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:39.5697493Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:39.5698386Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:39.5699390Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:39.5700145Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:39.5853592Z Running test: large_conv_model sync interval 2022-03-17T21:26:39.5853931Z 1 with iteration mapping 2022-03-17T21:26:39.5854152Z {0: 0, 1: 3} 2022-03-17T21:26:40.6752622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:42.8136549Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:42.8137360Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:42.8138408Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:42.8139174Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:42.8264053Z Running test: large_conv_model sync interval 2022-03-17T21:26:42.8264343Z 1 with iteration mapping 2022-03-17T21:26:42.8264633Z {0: 0, 1: 10} 2022-03-17T21:26:43.8924835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:53.5639474Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:53.5640570Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:53.5641640Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:26:53.5642683Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:26:53.5762490Z Running test: large_conv_model sync interval 2022-03-17T21:26:53.5763067Z 1 with iteration mapping 2022-03-17T21:26:53.5763551Z {0: 5, 1: 7} 2022-03-17T21:26:55.4237008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:26:55.4237529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:01.9020546Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:01.9021460Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:01.9022516Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:01.9023274Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:01.9123554Z Running test: large_conv_model sync interval 2022-03-17T21:27:01.9123882Z 1 with iteration mapping 2022-03-17T21:27:01.9124144Z {0: 5, 1: 8} 2022-03-17T21:27:03.0114184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:03.0114707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:10.5842099Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:10.5842938Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:10.5843985Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:10.5844735Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:10.5953643Z Running test: large_conv_model sync interval 2022-03-17T21:27:10.5953992Z 1 with iteration mapping 2022-03-17T21:27:10.5954248Z {0: 5, 1: 15} 2022-03-17T21:27:11.7346044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:11.7346819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:26.8804299Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:26.8805136Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:26.8806186Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:26.8807273Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:26.8917742Z Running test: small_model sync interval 2022-03-17T21:27:26.8918211Z 1 with iteration mapping 2022-03-17T21:27:26.8918627Z {0: 0, 1: 2} 2022-03-17T21:27:26.9042658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:26.9288084Z Running test: small_model sync interval 2022-03-17T21:27:26.9288772Z 1 with iteration mapping 2022-03-17T21:27:26.9289045Z {0: 0, 1: 3} 2022-03-17T21:27:26.9377730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:26.9616387Z Running test: small_model sync interval 2022-03-17T21:27:26.9616745Z 1 with iteration mapping 2022-03-17T21:27:26.9617015Z {0: 0, 1: 10} 2022-03-17T21:27:26.9708806Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.0209481Z Running test: small_model sync interval 2022-03-17T21:27:27.0209860Z 1 with iteration mapping 2022-03-17T21:27:27.0210111Z {0: 5, 1: 7} 2022-03-17T21:27:27.0295392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.0295865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.0668805Z Running test: small_model sync interval 2022-03-17T21:27:27.0669192Z 1 with iteration mapping 2022-03-17T21:27:27.0669443Z {0: 5, 1: 8} 2022-03-17T21:27:27.0754359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.0754857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.1159283Z Running test: small_model sync interval 2022-03-17T21:27:27.1159651Z 1 with iteration mapping 2022-03-17T21:27:27.1159909Z {0: 5, 1: 15} 2022-03-17T21:27:27.1247743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.1248245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.1962772Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.1963203Z 1 with iteration mapping 2022-03-17T21:27:27.1963486Z {0: 0, 1: 2} 2022-03-17T21:27:27.2044583Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2045404Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2046424Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2047182Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2076447Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.2076807Z 1 with iteration mapping 2022-03-17T21:27:27.2077064Z {0: 0, 1: 3} 2022-03-17T21:27:27.2172240Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2173472Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2174617Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2175356Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2199654Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.2200003Z 1 with iteration mapping 2022-03-17T21:27:27.2200277Z {0: 0, 1: 10} 2022-03-17T21:27:27.2408504Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2409702Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2410759Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2411517Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2435394Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.2435739Z 1 with iteration mapping 2022-03-17T21:27:27.2436014Z {0: 5, 1: 7} 2022-03-17T21:27:27.2465817Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:27:27.2597834Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2598632Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2599654Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2600426Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2625878Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.2626270Z 1 with iteration mapping 2022-03-17T21:27:27.2626523Z {0: 5, 1: 8} 2022-03-17T21:27:27.2798801Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2799590Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2800758Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.2801520Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.2831459Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.2831863Z 1 with iteration mapping 2022-03-17T21:27:27.2832120Z {0: 5, 1: 15} 2022-03-17T21:27:27.3122828Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3123606Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3124625Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3125401Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3150019Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.3150389Z 1 with iteration mapping 2022-03-17T21:27:27.3150667Z {0: 0, 1: 2} 2022-03-17T21:27:27.3179437Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:27:27.3226526Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3227293Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3228319Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3229083Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3253457Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.3253857Z 1 with iteration mapping 2022-03-17T21:27:27.3254294Z {0: 0, 1: 3} 2022-03-17T21:27:27.3342749Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3343531Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3344695Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3345474Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3369956Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.3370341Z 1 with iteration mapping 2022-03-17T21:27:27.3370598Z {0: 0, 1: 10} 2022-03-17T21:27:27.3582261Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3583551Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3584585Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3585367Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3608947Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.3609299Z 1 with iteration mapping 2022-03-17T21:27:27.3609579Z {0: 5, 1: 7} 2022-03-17T21:27:27.3771093Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3772221Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3773245Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3774010Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3798035Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.3798407Z 1 with iteration mapping 2022-03-17T21:27:27.3798674Z {0: 5, 1: 8} 2022-03-17T21:27:27.3974180Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3974970Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.3976158Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.3976910Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.4002359Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:27:27.4002719Z 1 with iteration mapping 2022-03-17T21:27:27.4003104Z {0: 5, 1: 15} 2022-03-17T21:27:27.4293598Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.4294386Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.4295419Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.4296168Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.4322197Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:27:27.4322574Z 1 with iteration mapping 2022-03-17T21:27:27.4322828Z {0: 0, 1: 2} 2022-03-17T21:27:27.4351730Z registered hook 2022-03-17T21:27:27.4352083Z registered hook 2022-03-17T21:27:27.4411542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.4585384Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:27:27.4585741Z 1 with iteration mapping 2022-03-17T21:27:27.4586017Z {0: 0, 1: 3} 2022-03-17T21:27:27.4612382Z registered hook 2022-03-17T21:27:27.4612809Z registered hook 2022-03-17T21:27:27.4673390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.4885945Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:27:27.4886384Z 1 with iteration mapping 2022-03-17T21:27:27.4886671Z {0: 0, 1: 10} 2022-03-17T21:27:27.4913041Z registered hook 2022-03-17T21:27:27.4913433Z registered hook 2022-03-17T21:27:27.4974510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.5470656Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:27:27.5471310Z 1 with iteration mapping 2022-03-17T21:27:27.5471778Z {0: 5, 1: 7} 2022-03-17T21:27:27.5498236Z registered hook 2022-03-17T21:27:27.5498645Z registered hook 2022-03-17T21:27:27.5556905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.5557837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.5936622Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:27:27.5937345Z 1 with iteration mapping 2022-03-17T21:27:27.5938272Z {0: 5, 1: 8} 2022-03-17T21:27:27.5965045Z registered hook 2022-03-17T21:27:27.5965773Z registered hook 2022-03-17T21:27:27.6023811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.6024759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.6445718Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:27:27.6446079Z 1 with iteration mapping 2022-03-17T21:27:27.6446356Z {0: 5, 1: 15} 2022-03-17T21:27:27.6473697Z registered hook 2022-03-17T21:27:27.6474092Z registered hook 2022-03-17T21:27:27.6532943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.6533433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.7228244Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:27:27.7228585Z 1 with iteration mapping 2022-03-17T21:27:27.7228862Z {0: 0, 1: 2} 2022-03-17T21:27:27.7255985Z registered hook 2022-03-17T21:27:27.7256358Z registered hook 2022-03-17T21:27:27.7272354Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T21:27:27.7273247Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 1 iterations. 2022-03-17T21:27:27.7319139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.7324222Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T21:27:27.7326715Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 1, total before compression 1000000, total after compression 2000, rate 500.0 2022-03-17T21:27:27.7492233Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:27:27.7492652Z 1 with iteration mapping 2022-03-17T21:27:27.7492919Z {0: 0, 1: 3} 2022-03-17T21:27:27.7520476Z registered hook 2022-03-17T21:27:27.7520860Z registered hook 2022-03-17T21:27:27.7566922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.7755857Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:27:27.7756217Z 1 with iteration mapping 2022-03-17T21:27:27.7756471Z {0: 0, 1: 10} 2022-03-17T21:27:27.7782906Z registered hook 2022-03-17T21:27:27.7783744Z registered hook 2022-03-17T21:27:27.7828147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.8210629Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:27:27.8211008Z 1 with iteration mapping 2022-03-17T21:27:27.8211285Z {0: 5, 1: 7} 2022-03-17T21:27:27.8238007Z registered hook 2022-03-17T21:27:27.8238599Z registered hook 2022-03-17T21:27:27.8281856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.8282364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.8575607Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:27:27.8576360Z 1 with iteration mapping 2022-03-17T21:27:27.8576594Z {0: 5, 1: 8} 2022-03-17T21:27:27.8602810Z registered hook 2022-03-17T21:27:27.8603203Z registered hook 2022-03-17T21:27:27.8647562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.8648056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.8972339Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:27:27.8972682Z 1 with iteration mapping 2022-03-17T21:27:27.8973205Z {0: 5, 1: 15} 2022-03-17T21:27:27.8999593Z registered hook 2022-03-17T21:27:27.9000014Z registered hook 2022-03-17T21:27:27.9044309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.9044800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.9580250Z Running test: batch_norm_net sync interval 2022-03-17T21:27:27.9580874Z 2 with iteration mapping 2022-03-17T21:27:27.9581147Z {0: 0, 1: 2} 2022-03-17T21:27:27.9644687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.9682575Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.9683372Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.9684403Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.9685175Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.9765544Z Running test: batch_norm_net sync interval 2022-03-17T21:27:27.9765888Z 2 with iteration mapping 2022-03-17T21:27:27.9766136Z {0: 0, 1: 3} 2022-03-17T21:27:27.9826273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:27.9882728Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.9883488Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.9884516Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:27.9885283Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:27.9967189Z Running test: batch_norm_net sync interval 2022-03-17T21:27:27.9967514Z 2 with iteration mapping 2022-03-17T21:27:27.9967785Z {0: 0, 1: 10} 2022-03-17T21:27:28.0028529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:28.0227413Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:28.0228194Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:28.0229360Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:28.0230125Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:28.0311536Z Running test: batch_norm_net sync interval 2022-03-17T21:27:28.0311860Z 2 with iteration mapping 2022-03-17T21:27:28.0312138Z {0: 5, 1: 7} 2022-03-17T21:27:28.0372040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:28.0372521Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:28.0509005Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:28.0509782Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:28.0510783Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:28.0511560Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:28.0594030Z Running test: batch_norm_net sync interval 2022-03-17T21:27:28.0594350Z 2 with iteration mapping 2022-03-17T21:27:28.0594617Z {0: 5, 1: 8} 2022-03-17T21:27:28.0652085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:28.0652576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:28.0811072Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:28.0811864Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:28.0812847Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:28.0813608Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:28.0894518Z Running test: batch_norm_net sync interval 2022-03-17T21:27:28.0894841Z 2 with iteration mapping 2022-03-17T21:27:28.0895109Z {0: 5, 1: 15} 2022-03-17T21:27:28.0952256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:28.0952896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:28.1251229Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:28.1252004Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:28.1253109Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:28.1253898Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:28.1333735Z Running test: large_conv_model sync interval 2022-03-17T21:27:28.1334059Z 3 with iteration mapping 2022-03-17T21:27:28.1334328Z {0: 0, 1: 2} 2022-03-17T21:27:29.2044762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:30.2910917Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:30.2911751Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:30.2912778Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:30.2913553Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:30.3019838Z Running test: large_conv_model sync interval 2022-03-17T21:27:30.3020186Z 3 with iteration mapping 2022-03-17T21:27:30.3020699Z {0: 0, 1: 3} 2022-03-17T21:27:31.3624892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:33.4959212Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:33.4960028Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:33.4961072Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:33.4961804Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:33.5070143Z Running test: large_conv_model sync interval 2022-03-17T21:27:33.5070505Z 3 with iteration mapping 2022-03-17T21:27:33.5070743Z {0: 0, 1: 10} 2022-03-17T21:27:34.5962381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:44.2446019Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:44.2447117Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:44.2448279Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:44.2449053Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:44.2559054Z Running test: large_conv_model sync interval 2022-03-17T21:27:44.2559438Z 3 with iteration mapping 2022-03-17T21:27:44.2559709Z {0: 5, 1: 7} 2022-03-17T21:27:45.3419720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:45.3420468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:51.8575288Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:51.8576115Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:51.8577138Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:27:51.8577902Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:27:51.8694934Z Running test: large_conv_model sync interval 2022-03-17T21:27:51.8695276Z 3 with iteration mapping 2022-03-17T21:27:51.8695526Z {0: 5, 1: 8} 2022-03-17T21:27:52.9612799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:27:52.9613319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:00.5227635Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:00.5228478Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:00.5229506Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:00.5230255Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:00.5322829Z Running test: large_conv_model sync interval 2022-03-17T21:28:00.5323182Z 3 with iteration mapping 2022-03-17T21:28:00.5323436Z {0: 5, 1: 15} 2022-03-17T21:28:01.6668334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:01.6668874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.7672173Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.7672949Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.7674280Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.7675049Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.7785933Z Running test: small_model sync interval 2022-03-17T21:28:16.7786249Z 4 with iteration mapping 2022-03-17T21:28:16.7786522Z {0: 0, 1: 2} 2022-03-17T21:28:16.7877605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.8021773Z Running test: small_model sync interval 2022-03-17T21:28:16.8022180Z 4 with iteration mapping 2022-03-17T21:28:16.8022462Z {0: 0, 1: 3} 2022-03-17T21:28:16.8109159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.8257348Z Running test: small_model sync interval 2022-03-17T21:28:16.8257761Z 4 with iteration mapping 2022-03-17T21:28:16.8258039Z {0: 0, 1: 10} 2022-03-17T21:28:16.8343475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.8611174Z Running test: small_model sync interval 2022-03-17T21:28:16.8611519Z 4 with iteration mapping 2022-03-17T21:28:16.8611776Z {0: 5, 1: 7} 2022-03-17T21:28:16.8697798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.8698298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.8924730Z Running test: small_model sync interval 2022-03-17T21:28:16.8925129Z 4 with iteration mapping 2022-03-17T21:28:16.8925364Z {0: 5, 1: 8} 2022-03-17T21:28:16.9012359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.9012853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.9243228Z Running test: small_model sync interval 2022-03-17T21:28:16.9243656Z 4 with iteration mapping 2022-03-17T21:28:16.9243880Z {0: 5, 1: 15} 2022-03-17T21:28:16.9329241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.9329666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:16.9684327Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:16.9684702Z 5 with iteration mapping 2022-03-17T21:28:16.9684960Z {0: 0, 1: 2} 2022-03-17T21:28:16.9753408Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.9754114Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.9755341Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.9756106Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.9780362Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:16.9780781Z 5 with iteration mapping 2022-03-17T21:28:16.9781272Z {0: 0, 1: 3} 2022-03-17T21:28:16.9854023Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.9854725Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.9855748Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:16.9856501Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:16.9881106Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:16.9881519Z 5 with iteration mapping 2022-03-17T21:28:16.9881787Z {0: 0, 1: 10} 2022-03-17T21:28:17.0033593Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0034310Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0035317Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0036072Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0061080Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.0061439Z 5 with iteration mapping 2022-03-17T21:28:17.0061697Z {0: 5, 1: 7} 2022-03-17T21:28:17.0178327Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0179082Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0180119Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0181388Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0206187Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.0206892Z 5 with iteration mapping 2022-03-17T21:28:17.0207174Z {0: 5, 1: 8} 2022-03-17T21:28:17.0335830Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0336630Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0337809Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0338597Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0364355Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.0364885Z 5 with iteration mapping 2022-03-17T21:28:17.0365344Z {0: 5, 1: 15} 2022-03-17T21:28:17.0558752Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0559480Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0560510Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0561265Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0586863Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.0587254Z 6 with iteration mapping 2022-03-17T21:28:17.0587509Z {0: 0, 1: 2} 2022-03-17T21:28:17.0664350Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0665133Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0666201Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0666965Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0680172Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.0680584Z 6 with iteration mapping 2022-03-17T21:28:17.0680828Z {0: 0, 1: 3} 2022-03-17T21:28:17.0754220Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0755128Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0756162Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0756914Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0781430Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.0781803Z 6 with iteration mapping 2022-03-17T21:28:17.0782058Z {0: 0, 1: 10} 2022-03-17T21:28:17.0930584Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0931316Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0932329Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.0933090Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.0958556Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.0958934Z 6 with iteration mapping 2022-03-17T21:28:17.0959209Z {0: 5, 1: 7} 2022-03-17T21:28:17.1078987Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.1079710Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.1080733Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.1081488Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.1106944Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.1107342Z 6 with iteration mapping 2022-03-17T21:28:17.1107598Z {0: 5, 1: 8} 2022-03-17T21:28:17.1237336Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.1238062Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.1239079Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.1239957Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.1265852Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:17.1266255Z 6 with iteration mapping 2022-03-17T21:28:17.1266501Z {0: 5, 1: 15} 2022-03-17T21:28:17.1469046Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.1469773Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.1470779Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:17.1471541Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:17.1496691Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:17.1497076Z 7 with iteration mapping 2022-03-17T21:28:17.1497342Z {0: 0, 1: 2} 2022-03-17T21:28:17.1584289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.1727289Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:17.1727647Z 7 with iteration mapping 2022-03-17T21:28:17.1727902Z {0: 0, 1: 3} 2022-03-17T21:28:17.1814902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.1966986Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:17.1967391Z 7 with iteration mapping 2022-03-17T21:28:17.1967671Z {0: 0, 1: 10} 2022-03-17T21:28:17.2053944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.2286805Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:17.2287171Z 7 with iteration mapping 2022-03-17T21:28:17.2287428Z {0: 5, 1: 7} 2022-03-17T21:28:17.2371856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.2372316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.2557556Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:17.2557931Z 7 with iteration mapping 2022-03-17T21:28:17.2558184Z {0: 5, 1: 8} 2022-03-17T21:28:17.2644217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.2644727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.2873952Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:17.2874328Z 7 with iteration mapping 2022-03-17T21:28:17.2874592Z {0: 5, 1: 15} 2022-03-17T21:28:17.2959755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.2960267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.3281076Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:17.3281490Z 8 with iteration mapping 2022-03-17T21:28:17.3281761Z {0: 0, 1: 2} 2022-03-17T21:28:17.3368141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.3508150Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:17.3508746Z 8 with iteration mapping 2022-03-17T21:28:17.3509253Z {0: 0, 1: 3} 2022-03-17T21:28:17.3595759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.3746424Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:17.3746790Z 8 with iteration mapping 2022-03-17T21:28:17.3747047Z {0: 0, 1: 10} 2022-03-17T21:28:17.3833643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.4082390Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:17.4082753Z 8 with iteration mapping 2022-03-17T21:28:17.4083022Z {0: 5, 1: 7} 2022-03-17T21:28:17.4168695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.4169202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.4353062Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:17.4353366Z 8 with iteration mapping 2022-03-17T21:28:17.4353647Z {0: 5, 1: 8} 2022-03-17T21:28:17.4439203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.4439718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.4631656Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:17.4632023Z 8 with iteration mapping 2022-03-17T21:28:17.4632300Z {0: 5, 1: 15} 2022-03-17T21:28:17.4720376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.4720890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.5009647Z Running test: batch_norm_net sync interval 2022-03-17T21:28:17.5009945Z 1 with iteration mapping 2022-03-17T21:28:17.5010231Z {0: 0, 1: 2} 2022-03-17T21:28:17.5046850Z Running test: batch_norm_net sync interval 2022-03-17T21:28:17.5047716Z 1 with iteration mapping 2022-03-17T21:28:17.5048298Z {0: 0, 1: 3} 2022-03-17T21:28:17.5081968Z Running test: batch_norm_net sync interval 2022-03-17T21:28:17.5082610Z 1 with iteration mapping 2022-03-17T21:28:17.5083141Z {0: 0, 1: 10} 2022-03-17T21:28:17.5116231Z Running test: batch_norm_net sync interval 2022-03-17T21:28:17.5116891Z 1 with iteration mapping 2022-03-17T21:28:17.5117416Z {0: 5, 1: 7} 2022-03-17T21:28:17.5186250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.5186787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.5317772Z Running test: batch_norm_net sync interval 2022-03-17T21:28:17.5318420Z 1 with iteration mapping 2022-03-17T21:28:17.5318942Z {0: 5, 1: 8} 2022-03-17T21:28:17.5386225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.5386712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.5515156Z Running test: batch_norm_net sync interval 2022-03-17T21:28:17.5515781Z 1 with iteration mapping 2022-03-17T21:28:17.5516303Z {0: 5, 1: 15} 2022-03-17T21:28:17.5583069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.5583592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:17.5715191Z Running test: large_conv_model sync interval 2022-03-17T21:28:17.5715800Z 1 with iteration mapping 2022-03-17T21:28:17.5716324Z {0: 0, 1: 2} 2022-03-17T21:28:17.5753120Z Running test: large_conv_model sync interval 2022-03-17T21:28:17.5753794Z 1 with iteration mapping 2022-03-17T21:28:17.5754314Z {0: 0, 1: 3} 2022-03-17T21:28:17.5789454Z Running test: large_conv_model sync interval 2022-03-17T21:28:17.5790102Z 1 with iteration mapping 2022-03-17T21:28:17.5790611Z {0: 0, 1: 10} 2022-03-17T21:28:17.5826015Z Running test: large_conv_model sync interval 2022-03-17T21:28:17.5826602Z 1 with iteration mapping 2022-03-17T21:28:17.5827115Z {0: 5, 1: 7} 2022-03-17T21:28:18.6747217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:18.6748134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:23.0540306Z Running test: large_conv_model sync interval 2022-03-17T21:28:23.0540838Z 1 with iteration mapping 2022-03-17T21:28:23.0541120Z {0: 5, 1: 8} 2022-03-17T21:28:24.1470997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:24.1471915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:28.5284508Z Running test: large_conv_model sync interval 2022-03-17T21:28:28.5285160Z 1 with iteration mapping 2022-03-17T21:28:28.5285708Z {0: 5, 1: 15} 2022-03-17T21:28:29.6329402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:29.6330307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.0198395Z Running test: small_model sync interval 2022-03-17T21:28:34.0198792Z 1 with iteration mapping 2022-03-17T21:28:34.0199071Z {0: 0, 1: 2} 2022-03-17T21:28:34.0242248Z Running test: small_model sync interval 2022-03-17T21:28:34.0242623Z 1 with iteration mapping 2022-03-17T21:28:34.0242883Z {0: 0, 1: 3} 2022-03-17T21:28:34.0285594Z Running test: small_model sync interval 2022-03-17T21:28:34.0286228Z 1 with iteration mapping 2022-03-17T21:28:34.0286696Z {0: 0, 1: 10} 2022-03-17T21:28:34.0329333Z Running test: small_model sync interval 2022-03-17T21:28:34.0329944Z 1 with iteration mapping 2022-03-17T21:28:34.0330467Z {0: 5, 1: 7} 2022-03-17T21:28:34.0422803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.0423286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.0604278Z Running test: small_model sync interval 2022-03-17T21:28:34.0604567Z 1 with iteration mapping 2022-03-17T21:28:34.0604857Z {0: 5, 1: 8} 2022-03-17T21:28:34.0697614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.0698535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.0881325Z Running test: small_model sync interval 2022-03-17T21:28:34.0881691Z 1 with iteration mapping 2022-03-17T21:28:34.0881944Z {0: 5, 1: 15} 2022-03-17T21:28:34.0970666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.0971111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.1150978Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1151654Z 1 with iteration mapping 2022-03-17T21:28:34.1151915Z {0: 0, 1: 2} 2022-03-17T21:28:34.1181585Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1181968Z 1 with iteration mapping 2022-03-17T21:28:34.1182201Z {0: 0, 1: 3} 2022-03-17T21:28:34.1212325Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1212708Z 1 with iteration mapping 2022-03-17T21:28:34.1212927Z {0: 0, 1: 10} 2022-03-17T21:28:34.1243092Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1243489Z 1 with iteration mapping 2022-03-17T21:28:34.1243756Z {0: 5, 1: 7} 2022-03-17T21:28:34.1378779Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1379119Z 1 with iteration mapping 2022-03-17T21:28:34.1379387Z {0: 5, 1: 8} 2022-03-17T21:28:34.1510978Z Running test: unjoined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1511389Z 1 with iteration mapping 2022-03-17T21:28:34.1511630Z {0: 5, 1: 15} 2022-03-17T21:28:34.1645001Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1645379Z 1 with iteration mapping 2022-03-17T21:28:34.1645635Z {0: 0, 1: 2} 2022-03-17T21:28:34.1674298Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1674638Z 1 with iteration mapping 2022-03-17T21:28:34.1674923Z {0: 0, 1: 3} 2022-03-17T21:28:34.1704505Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1704837Z 1 with iteration mapping 2022-03-17T21:28:34.1705105Z {0: 0, 1: 10} 2022-03-17T21:28:34.1734438Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1734837Z 1 with iteration mapping 2022-03-17T21:28:34.1735106Z {0: 5, 1: 7} 2022-03-17T21:28:34.1867447Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1867801Z 1 with iteration mapping 2022-03-17T21:28:34.1868078Z {0: 5, 1: 8} 2022-03-17T21:28:34.1995776Z Running test: joined_rank_with_unused_params_model sync interval 2022-03-17T21:28:34.1996117Z 1 with iteration mapping 2022-03-17T21:28:34.1996383Z {0: 5, 1: 15} 2022-03-17T21:28:34.2123041Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:34.2123382Z 1 with iteration mapping 2022-03-17T21:28:34.2123638Z {0: 0, 1: 2} 2022-03-17T21:28:34.2164492Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:34.2164841Z 1 with iteration mapping 2022-03-17T21:28:34.2165109Z {0: 0, 1: 3} 2022-03-17T21:28:34.2206840Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:34.2207157Z 1 with iteration mapping 2022-03-17T21:28:34.2207436Z {0: 0, 1: 10} 2022-03-17T21:28:34.2248431Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:34.2248799Z 1 with iteration mapping 2022-03-17T21:28:34.2249034Z {0: 5, 1: 7} 2022-03-17T21:28:34.2337543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.2338458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.2514919Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:34.2515280Z 1 with iteration mapping 2022-03-17T21:28:34.2515804Z {0: 5, 1: 8} 2022-03-17T21:28:34.2603181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.2603660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.2781789Z Running test: small_model_allreduce_hook sync interval 2022-03-17T21:28:34.2782163Z 1 with iteration mapping 2022-03-17T21:28:34.2782397Z {0: 5, 1: 15} 2022-03-17T21:28:34.2869464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.2870597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.3047595Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:34.3047907Z 1 with iteration mapping 2022-03-17T21:28:34.3048168Z {0: 0, 1: 2} 2022-03-17T21:28:34.3089304Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:34.3089650Z 1 with iteration mapping 2022-03-17T21:28:34.3089888Z {0: 0, 1: 3} 2022-03-17T21:28:34.3130431Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:34.3130772Z 1 with iteration mapping 2022-03-17T21:28:34.3131045Z {0: 0, 1: 10} 2022-03-17T21:28:34.3172256Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:34.3172603Z 1 with iteration mapping 2022-03-17T21:28:34.3172834Z {0: 5, 1: 7} 2022-03-17T21:28:34.3259865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.3261099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.3438360Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:34.3438681Z 1 with iteration mapping 2022-03-17T21:28:34.3438955Z {0: 5, 1: 8} 2022-03-17T21:28:34.3525777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.3526252Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.3708105Z Running test: small_model_power_sgd_hook sync interval 2022-03-17T21:28:34.3708453Z 1 with iteration mapping 2022-03-17T21:28:34.3708736Z {0: 5, 1: 15} 2022-03-17T21:28:34.3795858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.3796745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:34.9647961Z ok (123.775s) 2022-03-17T21:28:34.9648191Z 2022-03-17T21:28:34.9648562Z ---------------------------------------------------------------------- 2022-03-17T21:28:34.9648907Z Ran 1 test in 123.776s 2022-03-17T21:28:34.9649095Z 2022-03-17T21:28:34.9649189Z OK 2022-03-17T21:28:34.9652008Z 2022-03-17T21:28:34.9652472Z Generating XML reports... 2022-03-17T21:28:34.9691066Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212631.xml 2022-03-17T21:28:36.2705625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:28:36.2719804Z 2022-03-17T21:28:36.2720324Z Running tests... 2022-03-17T21:28:36.2720800Z ---------------------------------------------------------------------- 2022-03-17T21:28:37.8123088Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:28:37.8549763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68088 2022-03-17T21:28:37.8678725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68089 2022-03-17T21:28:38.9388256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:28:38.9389058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:28:38.9389846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:38.9390541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:38.9498079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:28:39.0402390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:28:40.6164772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:40.6165286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:40.6348133Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:40.6349072Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:40.6350265Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:40.6350997Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:40.6492113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:40.6492960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:28:40.6608226Z /opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py:1384: UserWarning: positional arguments and argument "destination" are deprecated. nn.Module.state_dict will not accept them in the future. Refer to https://pytorch.org/docs/master/generated/torch.nn.Module.html#torch.nn.Module.state_dict for details. 2022-03-17T21:28:40.6609066Z " and ".join(warn_msg) + " are deprecated. nn.Module.state_dict will not accept them in the future. " 2022-03-17T21:28:40.9760392Z ok (4.704s) 2022-03-17T21:28:40.9760614Z 2022-03-17T21:28:40.9761052Z ---------------------------------------------------------------------- 2022-03-17T21:28:40.9761371Z Ran 1 test in 4.704s 2022-03-17T21:28:40.9761530Z 2022-03-17T21:28:40.9761599Z OK 2022-03-17T21:28:40.9761741Z 2022-03-17T21:28:40.9761877Z Generating XML reports... 2022-03-17T21:28:40.9803195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212836.xml 2022-03-17T21:28:42.2821142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:28:42.2835419Z 2022-03-17T21:28:42.2835826Z Running tests... 2022-03-17T21:28:42.2836534Z ---------------------------------------------------------------------- 2022-03-17T21:28:43.8153253Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:28:43.8582979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68209 2022-03-17T21:28:43.8710544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68210 2022-03-17T21:28:44.9689844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:28:44.9690656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:28:44.9691887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:44.9692602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:44.9699068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:28:44.9699805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:28:46.8787035Z ok (4.595s) 2022-03-17T21:28:46.8787262Z 2022-03-17T21:28:46.8787701Z ---------------------------------------------------------------------- 2022-03-17T21:28:46.8788293Z Ran 1 test in 4.595s 2022-03-17T21:28:46.8788471Z 2022-03-17T21:28:46.8788564Z OK 2022-03-17T21:28:46.8788669Z 2022-03-17T21:28:46.8788765Z Generating XML reports... 2022-03-17T21:28:46.8833740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212842.xml 2022-03-17T21:28:48.1691973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:28:48.1705994Z 2022-03-17T21:28:48.1706412Z Running tests... 2022-03-17T21:28:48.1706883Z ---------------------------------------------------------------------- 2022-03-17T21:28:49.6919387Z test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:28:49.7343421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68330 2022-03-17T21:28:49.7471321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68331 2022-03-17T21:28:50.8592227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:28:50.8592765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:28:50.8593504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:50.8594212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:50.8702293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:28:50.9603825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:28:50.9813964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:28:50.9814520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:28:50.9815213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:28:50.9815909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:28:51.2521244Z ok (3.081s) 2022-03-17T21:28:51.2521456Z 2022-03-17T21:28:51.2521850Z ---------------------------------------------------------------------- 2022-03-17T21:28:51.2522189Z Ran 1 test in 3.081s 2022-03-17T21:28:51.2522369Z 2022-03-17T21:28:51.2522443Z OK 2022-03-17T21:28:51.2522569Z 2022-03-17T21:28:51.2522683Z Generating XML reports... 2022-03-17T21:28:51.2564535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212848.xml 2022-03-17T21:28:52.4822711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:28:52.4837891Z 2022-03-17T21:28:52.4838168Z Running tests... 2022-03-17T21:28:52.4838621Z ---------------------------------------------------------------------- 2022-03-17T21:28:54.0098101Z test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:28:54.0529372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68451 2022-03-17T21:28:54.0657056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68452 2022-03-17T21:28:55.1258538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:28:55.1259572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:28:55.1261144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:55.1261890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:55.1366415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:28:55.2271255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:28:55.2481237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:28:55.2481793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:28:55.2482473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:28:55.2483175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:28:55.4705927Z ok (2.986s) 2022-03-17T21:28:55.4706105Z 2022-03-17T21:28:55.4706547Z ---------------------------------------------------------------------- 2022-03-17T21:28:55.4706979Z Ran 1 test in 2.987s 2022-03-17T21:28:55.4707256Z 2022-03-17T21:28:55.4707368Z OK 2022-03-17T21:28:55.4707515Z 2022-03-17T21:28:55.4707625Z Generating XML reports... 2022-03-17T21:28:55.4751639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212852.xml 2022-03-17T21:28:56.7055914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:28:56.7071350Z 2022-03-17T21:28:56.7071600Z Running tests... 2022-03-17T21:28:56.7072044Z ---------------------------------------------------------------------- 2022-03-17T21:28:58.2387636Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:28:58.2818997Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68572 2022-03-17T21:28:58.2949457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68573 2022-03-17T21:28:59.4144131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:28:59.4144659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:28:59.4145442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:59.4146126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:28:59.4152745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:28:59.4153479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:01.0414523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:29:01.0415072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:29:01.0492853Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:29:01.0494798Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:29:01.4027488Z ok (4.695s) 2022-03-17T21:29:01.4027792Z 2022-03-17T21:29:01.4028307Z ---------------------------------------------------------------------- 2022-03-17T21:29:01.4028605Z Ran 1 test in 4.696s 2022-03-17T21:29:01.4028772Z 2022-03-17T21:29:01.4028865Z OK 2022-03-17T21:29:01.4029001Z 2022-03-17T21:29:01.4029136Z Generating XML reports... 2022-03-17T21:29:01.4071029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212856.xml 2022-03-17T21:29:02.7054341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:02.7069004Z 2022-03-17T21:29:02.7069248Z Running tests... 2022-03-17T21:29:02.7069721Z ---------------------------------------------------------------------- 2022-03-17T21:29:04.2462723Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:04.2891700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68693 2022-03-17T21:29:04.3021387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68694 2022-03-17T21:29:05.4038565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:05.4039120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:05.4039871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:05.4040568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:05.4047291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:05.4047811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:07.0297362Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:29:07.0495393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:29:07.0495902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:29:07.4100865Z ok (4.703s) 2022-03-17T21:29:07.4101416Z 2022-03-17T21:29:07.4101832Z ---------------------------------------------------------------------- 2022-03-17T21:29:07.4102178Z Ran 1 test in 4.703s 2022-03-17T21:29:07.4102324Z 2022-03-17T21:29:07.4102443Z OK 2022-03-17T21:29:07.4102580Z 2022-03-17T21:29:07.4102716Z Generating XML reports... 2022-03-17T21:29:07.4146389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212902.xml 2022-03-17T21:29:08.7101667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:08.7116354Z 2022-03-17T21:29:08.7116569Z Running tests... 2022-03-17T21:29:08.7117019Z ---------------------------------------------------------------------- 2022-03-17T21:29:10.2579555Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:10.3008977Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68814 2022-03-17T21:29:10.3136950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68815 2022-03-17T21:29:11.4330167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:11.4330726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:11.4331487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:11.4332193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:11.4338225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:11.4338706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:11.6184227Z ok (2.906s) 2022-03-17T21:29:11.6184491Z 2022-03-17T21:29:11.6184880Z ---------------------------------------------------------------------- 2022-03-17T21:29:11.6185218Z Ran 1 test in 2.907s 2022-03-17T21:29:11.6185389Z 2022-03-17T21:29:11.6185463Z OK 2022-03-17T21:29:11.6185605Z 2022-03-17T21:29:11.6185758Z Generating XML reports... 2022-03-17T21:29:11.6227734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212908.xml 2022-03-17T21:29:12.9147636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:12.9161683Z 2022-03-17T21:29:12.9161919Z Running tests... 2022-03-17T21:29:12.9162365Z ---------------------------------------------------------------------- 2022-03-17T21:29:14.4638097Z test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:14.5069294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68929 2022-03-17T21:29:14.5196997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68930 2022-03-17T21:29:15.6038927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:15.6039537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:15.6040327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:15.6041005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:15.6148783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:15.7050563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:15.9246017Z ok (3.008s) 2022-03-17T21:29:15.9246272Z 2022-03-17T21:29:15.9246666Z ---------------------------------------------------------------------- 2022-03-17T21:29:15.9247010Z Ran 1 test in 3.008s 2022-03-17T21:29:15.9247158Z 2022-03-17T21:29:15.9247267Z OK 2022-03-17T21:29:15.9247408Z 2022-03-17T21:29:15.9247544Z Generating XML reports... 2022-03-17T21:29:15.9291984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212912.xml 2022-03-17T21:29:17.2096461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:17.2111006Z 2022-03-17T21:29:17.2111429Z Running tests... 2022-03-17T21:29:17.2111851Z ---------------------------------------------------------------------- 2022-03-17T21:29:18.7668131Z test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:18.8098999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69048 2022-03-17T21:29:18.8232346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69049 2022-03-17T21:29:19.9281763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:19.9282347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:19.9283360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:19.9284061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:19.9390386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:20.0295267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:20.2282169Z ok (3.017s) 2022-03-17T21:29:20.2282586Z 2022-03-17T21:29:20.2283210Z ---------------------------------------------------------------------- 2022-03-17T21:29:20.2283811Z Ran 1 test in 3.017s 2022-03-17T21:29:20.2284111Z 2022-03-17T21:29:20.2284271Z OK 2022-03-17T21:29:20.2284546Z 2022-03-17T21:29:20.2284762Z Generating XML reports... 2022-03-17T21:29:20.2328781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212917.xml 2022-03-17T21:29:21.5150514Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:21.5164560Z 2022-03-17T21:29:21.5164830Z Running tests... 2022-03-17T21:29:21.5165231Z ---------------------------------------------------------------------- 2022-03-17T21:29:21.5185886Z test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:29:21.5186163Z 2022-03-17T21:29:21.5186444Z ---------------------------------------------------------------------- 2022-03-17T21:29:21.5186788Z Ran 1 test in 0.002s 2022-03-17T21:29:21.5186922Z 2022-03-17T21:29:21.5187041Z OK (skipped=1) 2022-03-17T21:29:21.5187161Z 2022-03-17T21:29:21.5187289Z Generating XML reports... 2022-03-17T21:29:21.5231495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212921.xml 2022-03-17T21:29:22.6653107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:22.6667444Z 2022-03-17T21:29:22.6667706Z Running tests... 2022-03-17T21:29:22.6668141Z ---------------------------------------------------------------------- 2022-03-17T21:29:24.2124503Z test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:24.2554797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69198 2022-03-17T21:29:24.2684450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69199 2022-03-17T21:29:25.3875696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:25.3876232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:25.3877006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:25.3877825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:25.3987265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:25.4887199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:25.5004517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:29:25.5005103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:29:25.5006048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:25.5006846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:25.7735793Z ok (3.107s) 2022-03-17T21:29:25.7735987Z 2022-03-17T21:29:25.7736638Z ---------------------------------------------------------------------- 2022-03-17T21:29:25.7736985Z Ran 1 test in 3.107s 2022-03-17T21:29:25.7737154Z 2022-03-17T21:29:25.7737259Z OK 2022-03-17T21:29:25.7737366Z 2022-03-17T21:29:25.7737529Z Generating XML reports... 2022-03-17T21:29:25.7781961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212922.xml 2022-03-17T21:29:27.0546251Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:27.0560720Z 2022-03-17T21:29:27.0560948Z Running tests... 2022-03-17T21:29:27.0561396Z ---------------------------------------------------------------------- 2022-03-17T21:29:28.5991487Z test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:28.6422621Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69323 2022-03-17T21:29:28.6550403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69324 2022-03-17T21:29:29.7517514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:29.7518054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:29.7518819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:29.7519522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:29.7628533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:29.8531680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:30.0601895Z skip (3.004s) 2022-03-17T21:29:30.0602078Z 2022-03-17T21:29:30.0602484Z ---------------------------------------------------------------------- 2022-03-17T21:29:30.0602820Z Ran 1 test in 3.004s 2022-03-17T21:29:30.0603003Z 2022-03-17T21:29:30.0603119Z OK (skipped=1) 2022-03-17T21:29:30.0603279Z 2022-03-17T21:29:30.0603388Z Generating XML reports... 2022-03-17T21:29:30.0647587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212927.xml 2022-03-17T21:29:31.3463917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:31.3478869Z 2022-03-17T21:29:31.3479325Z Running tests... 2022-03-17T21:29:31.3479844Z ---------------------------------------------------------------------- 2022-03-17T21:29:32.8960053Z test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:32.9392664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69438 2022-03-17T21:29:32.9520334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69439 2022-03-17T21:29:34.0368404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:34.0368964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:34.0369733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:34.0370663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:34.0376906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:34.0377355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:34.2567420Z ok (2.908s) 2022-03-17T21:29:34.2567674Z 2022-03-17T21:29:34.2568292Z ---------------------------------------------------------------------- 2022-03-17T21:29:34.2568615Z Ran 1 test in 2.909s 2022-03-17T21:29:34.2568774Z 2022-03-17T21:29:34.2568854Z OK 2022-03-17T21:29:34.2569259Z 2022-03-17T21:29:34.2569403Z Generating XML reports... 2022-03-17T21:29:34.2611712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212931.xml 2022-03-17T21:29:35.5391861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:35.5405818Z 2022-03-17T21:29:35.5406050Z Running tests... 2022-03-17T21:29:35.5406476Z ---------------------------------------------------------------------- 2022-03-17T21:29:37.0850886Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:37.1282424Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69553 2022-03-17T21:29:37.1409843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69554 2022-03-17T21:29:38.2716143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:38.2716695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:38.2717492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:38.2718179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:38.2724562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:38.2725632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:38.3040737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:29:38.3041275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:29:38.3041949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:38.3042623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:38.3188524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:29:38.3190455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:29:38.3191141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:29:38.3291010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:29:38.3414271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1 2022-03-17T21:29:38.3415336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0 2022-03-17T21:29:38.3416612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:29:38.3516926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes. 2022-03-17T21:29:38.5457787Z ok (3.005s) 2022-03-17T21:29:38.5458018Z 2022-03-17T21:29:38.5458437Z ---------------------------------------------------------------------- 2022-03-17T21:29:38.5458757Z Ran 1 test in 3.005s 2022-03-17T21:29:38.5458922Z 2022-03-17T21:29:38.5459006Z OK 2022-03-17T21:29:38.5459123Z 2022-03-17T21:29:38.5459240Z Generating XML reports... 2022-03-17T21:29:38.5501636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212935.xml 2022-03-17T21:29:39.8336272Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:39.8351699Z 2022-03-17T21:29:39.8352323Z Running tests... 2022-03-17T21:29:39.8352821Z ---------------------------------------------------------------------- 2022-03-17T21:29:41.3821950Z test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:41.4253590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69692 2022-03-17T21:29:41.4382290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69693 2022-03-17T21:29:42.5287635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:42.5288171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:42.5288954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:42.5289645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:42.5295569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:42.5296284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:42.5403311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:29:42.5403824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:29:42.5404540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:42.5405152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:42.8430636Z ok (3.008s) 2022-03-17T21:29:42.8430818Z 2022-03-17T21:29:42.8431175Z ---------------------------------------------------------------------- 2022-03-17T21:29:42.8431523Z Ran 1 test in 3.008s 2022-03-17T21:29:42.8431703Z 2022-03-17T21:29:42.8431795Z OK 2022-03-17T21:29:42.8431893Z 2022-03-17T21:29:42.8431981Z Generating XML reports... 2022-03-17T21:29:42.8475829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212939.xml 2022-03-17T21:29:44.1205525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:44.1219652Z 2022-03-17T21:29:44.1219904Z Running tests... 2022-03-17T21:29:44.1220764Z ---------------------------------------------------------------------- 2022-03-17T21:29:45.6627374Z test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:45.7053135Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69813 2022-03-17T21:29:45.7181356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69814 2022-03-17T21:29:46.8333473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:46.8334040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:46.8334774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:46.8335687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:46.8342392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:46.8342915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:47.0227940Z ok (2.900s) 2022-03-17T21:29:47.0228210Z 2022-03-17T21:29:47.0228614Z ---------------------------------------------------------------------- 2022-03-17T21:29:47.0228940Z Ran 1 test in 2.901s 2022-03-17T21:29:47.0229172Z 2022-03-17T21:29:47.0229248Z OK 2022-03-17T21:29:47.0229486Z 2022-03-17T21:29:47.0229991Z Generating XML reports... 2022-03-17T21:29:47.0271286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212944.xml 2022-03-17T21:29:48.3161406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:48.3175039Z 2022-03-17T21:29:48.3175421Z Running tests... 2022-03-17T21:29:48.3176382Z ---------------------------------------------------------------------- 2022-03-17T21:29:49.8710996Z test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:49.9141307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69928 2022-03-17T21:29:49.9270872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69929 2022-03-17T21:29:51.0031117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:51.0031823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:51.0032623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:51.0033297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:51.0139977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:51.1042508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:51.4321203Z ok (3.114s) 2022-03-17T21:29:51.4321402Z 2022-03-17T21:29:51.4321782Z ---------------------------------------------------------------------- 2022-03-17T21:29:51.4322127Z Ran 1 test in 3.115s 2022-03-17T21:29:51.4322283Z 2022-03-17T21:29:51.4322813Z OK 2022-03-17T21:29:51.4323020Z 2022-03-17T21:29:51.4323154Z Generating XML reports... 2022-03-17T21:29:51.4365502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212948.xml 2022-03-17T21:29:52.7135237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:52.7150210Z 2022-03-17T21:29:52.7150633Z Running tests... 2022-03-17T21:29:52.7151135Z ---------------------------------------------------------------------- 2022-03-17T21:29:54.2382840Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:54.2809555Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70043 2022-03-17T21:29:54.2939497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70044 2022-03-17T21:29:55.3812180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:55.3813173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:55.3813968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:55.3814640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:55.3922854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:55.4825675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:55.4939998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:29:55.4940778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:29:55.4941497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:55.4942346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:55.7989396Z ok (3.084s) 2022-03-17T21:29:55.7989573Z 2022-03-17T21:29:55.7990025Z ---------------------------------------------------------------------- 2022-03-17T21:29:55.7990386Z Ran 1 test in 3.084s 2022-03-17T21:29:55.7990498Z 2022-03-17T21:29:55.7990572Z OK 2022-03-17T21:29:55.7990707Z 2022-03-17T21:29:55.7990842Z Generating XML reports... 2022-03-17T21:29:55.8034107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212952.xml 2022-03-17T21:29:57.0575099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:29:57.0589737Z 2022-03-17T21:29:57.0589895Z Running tests... 2022-03-17T21:29:57.0590334Z ---------------------------------------------------------------------- 2022-03-17T21:29:58.6032282Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:29:58.6461778Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70164 2022-03-17T21:29:58.6589194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70165 2022-03-17T21:29:59.7431356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:29:59.7431886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:29:59.7432687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:59.7433377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:29:59.7541987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:29:59.8443096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:29:59.8652672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:29:59.8653409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:29:59.8654133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:29:59.8654823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:00.1640093Z ok (3.105s) 2022-03-17T21:30:00.1640419Z 2022-03-17T21:30:00.1641026Z ---------------------------------------------------------------------- 2022-03-17T21:30:00.1641617Z Ran 1 test in 3.105s 2022-03-17T21:30:00.1641926Z 2022-03-17T21:30:00.1642099Z OK 2022-03-17T21:30:00.1642337Z 2022-03-17T21:30:00.1642578Z Generating XML reports... 2022-03-17T21:30:00.1685303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212957.xml 2022-03-17T21:30:01.4567065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:01.4580946Z 2022-03-17T21:30:01.4581278Z Running tests... 2022-03-17T21:30:01.4582262Z ---------------------------------------------------------------------- 2022-03-17T21:30:03.0086040Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:30:03.0529059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70285 2022-03-17T21:30:03.0662445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70286 2022-03-17T21:30:04.1222383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:04.1222907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:04.1223919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:04.1224631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:04.1330599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:04.2237071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:06.0754958Z ok (4.617s) 2022-03-17T21:30:06.0755177Z 2022-03-17T21:30:06.0755558Z ---------------------------------------------------------------------- 2022-03-17T21:30:06.0755902Z Ran 1 test in 4.617s 2022-03-17T21:30:06.0756061Z 2022-03-17T21:30:06.0756142Z OK 2022-03-17T21:30:06.0756254Z 2022-03-17T21:30:06.0756387Z Generating XML reports... 2022-03-17T21:30:06.0797980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213001.xml 2022-03-17T21:30:07.3774061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:07.3789458Z 2022-03-17T21:30:07.3789833Z Running tests... 2022-03-17T21:30:07.3790274Z ---------------------------------------------------------------------- 2022-03-17T21:30:08.9193228Z test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:30:08.9627747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70406 2022-03-17T21:30:08.9757280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70407 2022-03-17T21:30:10.0799798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:10.0800381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:10.0801201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:10.0801876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:10.0910768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:10.1810876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:10.3804358Z ok (3.001s) 2022-03-17T21:30:10.3804571Z 2022-03-17T21:30:10.3804969Z ---------------------------------------------------------------------- 2022-03-17T21:30:10.3805310Z Ran 1 test in 3.001s 2022-03-17T21:30:10.3805457Z 2022-03-17T21:30:10.3805554Z OK 2022-03-17T21:30:10.3805689Z 2022-03-17T21:30:10.3805820Z Generating XML reports... 2022-03-17T21:30:10.3848744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213007.xml 2022-03-17T21:30:11.6702022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:11.6716895Z 2022-03-17T21:30:11.6717039Z Running tests... 2022-03-17T21:30:11.6717760Z ---------------------------------------------------------------------- 2022-03-17T21:30:13.2270409Z test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:30:13.2697931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70521 2022-03-17T21:30:13.2831109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70522 2022-03-17T21:30:14.4002433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:14.4003025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:14.4003728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:14.4004731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:14.4110747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:14.5014350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:14.6879169Z ok (3.016s) 2022-03-17T21:30:14.6879399Z 2022-03-17T21:30:14.6879799Z ---------------------------------------------------------------------- 2022-03-17T21:30:14.6880144Z Ran 1 test in 3.016s 2022-03-17T21:30:14.6880291Z 2022-03-17T21:30:14.6880386Z OK 2022-03-17T21:30:14.6880479Z 2022-03-17T21:30:14.6880590Z Generating XML reports... 2022-03-17T21:30:14.6922714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213011.xml 2022-03-17T21:30:15.9758364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:15.9772196Z 2022-03-17T21:30:15.9772532Z Running tests... 2022-03-17T21:30:15.9772945Z ---------------------------------------------------------------------- 2022-03-17T21:30:17.5103850Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:30:17.5533457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70636 2022-03-17T21:30:17.5663006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70637 2022-03-17T21:30:18.6596343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:18.6597233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:18.6598007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:18.6598711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:18.6704641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:18.7607950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:19.0713380Z ok (3.094s) 2022-03-17T21:30:19.0713812Z 2022-03-17T21:30:19.0714391Z ---------------------------------------------------------------------- 2022-03-17T21:30:19.0714715Z Ran 1 test in 3.094s 2022-03-17T21:30:19.0714882Z 2022-03-17T21:30:19.0714984Z OK 2022-03-17T21:30:19.0715030Z 2022-03-17T21:30:19.0716173Z Generating XML reports... 2022-03-17T21:30:19.0759107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213015.xml 2022-03-17T21:30:20.3543569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:20.3558693Z 2022-03-17T21:30:20.3559068Z Running tests... 2022-03-17T21:30:20.3559583Z ---------------------------------------------------------------------- 2022-03-17T21:30:21.8926545Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:30:21.9357691Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70755 2022-03-17T21:30:21.9486582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70756 2022-03-17T21:30:23.0471757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:23.0472304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:23.0473019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:23.0473708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:23.0481027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:23.0481477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:23.3535166Z ok (2.997s) 2022-03-17T21:30:23.3535529Z 2022-03-17T21:30:23.3536034Z ---------------------------------------------------------------------- 2022-03-17T21:30:23.3536358Z Ran 1 test in 2.998s 2022-03-17T21:30:23.3536441Z 2022-03-17T21:30:23.3536532Z OK 2022-03-17T21:30:23.3536668Z 2022-03-17T21:30:23.3536782Z Generating XML reports... 2022-03-17T21:30:23.3578306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213020.xml 2022-03-17T21:30:24.6330249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:24.6345747Z 2022-03-17T21:30:24.6346051Z Running tests... 2022-03-17T21:30:24.6346503Z ---------------------------------------------------------------------- 2022-03-17T21:30:26.1628895Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:30:26.2055677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70874 2022-03-17T21:30:26.2185629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70875 2022-03-17T21:30:27.3255489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:27.3256031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:27.3256827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:27.3257519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:27.3366052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:27.3369211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:30:27.4266445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:27.4270750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:30:27.4271462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:27.4283762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:27.4392005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:30:27.4392510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:30:27.4393188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:30:27.4393858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:30:29.8975485Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:30:45.4547097Z ok (20.820s) 2022-03-17T21:30:45.4547404Z 2022-03-17T21:30:45.4547808Z ---------------------------------------------------------------------- 2022-03-17T21:30:45.4548177Z Ran 1 test in 20.820s 2022-03-17T21:30:45.4548308Z 2022-03-17T21:30:45.4548408Z OK 2022-03-17T21:30:45.4548523Z 2022-03-17T21:30:45.4548657Z Generating XML reports... 2022-03-17T21:30:45.4591435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213024.xml 2022-03-17T21:30:46.7572002Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:30:46.7586830Z 2022-03-17T21:30:46.7587000Z Running tests... 2022-03-17T21:30:46.7587451Z ---------------------------------------------------------------------- 2022-03-17T21:30:48.2938623Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:30:48.3368711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71007 2022-03-17T21:30:48.3496092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71008 2022-03-17T21:30:49.4756047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:30:49.4756570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:30:49.4757353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:49.4758041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:30:49.4866718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:30:49.4870044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:30:49.5769285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:30:49.5773562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:30:49.5774268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:49.5789013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:30:49.5983352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:30:49.5983943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:30:49.5984634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:30:49.5985323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:30:52.0463502Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:30:52.0464029Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:31:07.6838989Z ok (20.925s) 2022-03-17T21:31:07.6839201Z 2022-03-17T21:31:07.6839607Z ---------------------------------------------------------------------- 2022-03-17T21:31:07.6839908Z Ran 1 test in 20.925s 2022-03-17T21:31:07.6840019Z 2022-03-17T21:31:07.6840110Z OK 2022-03-17T21:31:07.6840246Z 2022-03-17T21:31:07.6840362Z Generating XML reports... 2022-03-17T21:31:07.6882821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213046.xml 2022-03-17T21:31:08.9864764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:08.9879837Z 2022-03-17T21:31:08.9880061Z Running tests... 2022-03-17T21:31:08.9880847Z ---------------------------------------------------------------------- 2022-03-17T21:31:10.5377446Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:31:10.5822889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71140 2022-03-17T21:31:10.5953389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71141 2022-03-17T21:31:11.7161000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:11.7161713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:11.7162505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:11.7163201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:11.7169477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:11.7169936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:11.8999274Z skip (2.912s) 2022-03-17T21:31:11.8999488Z 2022-03-17T21:31:11.8999874Z ---------------------------------------------------------------------- 2022-03-17T21:31:11.9000206Z Ran 1 test in 2.912s 2022-03-17T21:31:11.9000362Z 2022-03-17T21:31:11.9000467Z OK (skipped=1) 2022-03-17T21:31:11.9000578Z 2022-03-17T21:31:11.9000688Z Generating XML reports... 2022-03-17T21:31:11.9043174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213108.xml 2022-03-17T21:31:13.1482925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:13.1497807Z 2022-03-17T21:31:13.1497927Z Running tests... 2022-03-17T21:31:13.1498664Z ---------------------------------------------------------------------- 2022-03-17T21:31:14.6983405Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:31:14.7414070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71255 2022-03-17T21:31:14.7546532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71256 2022-03-17T21:31:15.8560830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:15.8561399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:15.8562197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:15.8562873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:15.8569344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:15.8569824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:17.8680353Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms 2022-03-17T21:31:18.1623746Z ok (5.012s) 2022-03-17T21:31:18.1623953Z 2022-03-17T21:31:18.1624348Z ---------------------------------------------------------------------- 2022-03-17T21:31:18.1624670Z Ran 1 test in 5.013s 2022-03-17T21:31:18.1624834Z 2022-03-17T21:31:18.1624907Z OK 2022-03-17T21:31:18.1625039Z 2022-03-17T21:31:18.1625170Z Generating XML reports... 2022-03-17T21:31:18.1668164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213113.xml 2022-03-17T21:31:19.4775244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:19.4791271Z 2022-03-17T21:31:19.4791381Z Running tests... 2022-03-17T21:31:19.4792163Z ---------------------------------------------------------------------- 2022-03-17T21:31:21.0389089Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:31:21.0827922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71370 2022-03-17T21:31:21.0964244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71371 2022-03-17T21:31:22.1879213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:22.1879956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:22.1880760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:22.1881442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:22.1989882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:22.2890946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:22.3007168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:31:22.3007685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:31:22.3008373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:31:22.3009079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:31:22.3011826Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms. 2022-03-17T21:31:22.3012233Z No ranks successfully processed in monitoredBarrier. 2022-03-17T21:31:22.3043175Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms 2022-03-17T21:31:22.6015286Z ok (3.122s) 2022-03-17T21:31:22.6015510Z 2022-03-17T21:31:22.6016240Z ---------------------------------------------------------------------- 2022-03-17T21:31:22.6016620Z Ran 1 test in 3.122s 2022-03-17T21:31:22.6016786Z 2022-03-17T21:31:22.6016869Z OK 2022-03-17T21:31:22.6017018Z 2022-03-17T21:31:22.6017137Z Generating XML reports... 2022-03-17T21:31:22.6060122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213119.xml 2022-03-17T21:31:23.8846719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:23.8861351Z 2022-03-17T21:31:23.8861481Z Running tests... 2022-03-17T21:31:23.8861945Z ---------------------------------------------------------------------- 2022-03-17T21:31:25.4206177Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:31:25.4632724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71491 2022-03-17T21:31:25.4760300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71492 2022-03-17T21:31:26.5425362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:26.5426067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:26.5426846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:26.5427561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:26.5533973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:26.6437765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:26.6647909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:31:26.6648689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:31:26.6649374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:31:26.6650070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:31:26.7756981Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms 2022-03-17T21:31:27.0810741Z ok (3.195s) 2022-03-17T21:31:27.0810939Z 2022-03-17T21:31:27.0811563Z ---------------------------------------------------------------------- 2022-03-17T21:31:27.0811912Z Ran 1 test in 3.195s 2022-03-17T21:31:27.0812078Z 2022-03-17T21:31:27.0812176Z OK 2022-03-17T21:31:27.0812325Z 2022-03-17T21:31:27.0812493Z Generating XML reports... 2022-03-17T21:31:27.0853836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213123.xml 2022-03-17T21:31:28.3893004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:28.3908411Z 2022-03-17T21:31:28.3908854Z Running tests... 2022-03-17T21:31:28.3909371Z ---------------------------------------------------------------------- 2022-03-17T21:31:29.9310817Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:31:29.9745648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71612 2022-03-17T21:31:29.9872198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71613 2022-03-17T21:31:31.0429160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:31.0429725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:31.0430455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:31.0431162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:31.0537938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:31.1444701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:31.2918966Z skip (2.901s) 2022-03-17T21:31:31.2919167Z 2022-03-17T21:31:31.2919539Z ---------------------------------------------------------------------- 2022-03-17T21:31:31.2919890Z Ran 1 test in 2.901s 2022-03-17T21:31:31.2920045Z 2022-03-17T21:31:31.2920150Z OK (skipped=1) 2022-03-17T21:31:31.2920309Z 2022-03-17T21:31:31.2920430Z Generating XML reports... 2022-03-17T21:31:31.2963172Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213128.xml 2022-03-17T21:31:32.5807118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:32.5821076Z 2022-03-17T21:31:32.5821458Z Running tests... 2022-03-17T21:31:32.5821980Z ---------------------------------------------------------------------- 2022-03-17T21:31:32.5849741Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:31:32.5850049Z 2022-03-17T21:31:32.5850373Z ---------------------------------------------------------------------- 2022-03-17T21:31:32.5850691Z Ran 1 test in 0.003s 2022-03-17T21:31:32.5850860Z 2022-03-17T21:31:32.5850961Z OK (skipped=1) 2022-03-17T21:31:32.5851119Z 2022-03-17T21:31:32.5851253Z Generating XML reports... 2022-03-17T21:31:32.5895277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213132.xml 2022-03-17T21:31:33.7287357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:33.7301587Z 2022-03-17T21:31:33.7301844Z Running tests... 2022-03-17T21:31:33.7302274Z ---------------------------------------------------------------------- 2022-03-17T21:31:33.7331366Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:31:33.7331697Z 2022-03-17T21:31:33.7332163Z ---------------------------------------------------------------------- 2022-03-17T21:31:33.7332519Z Ran 1 test in 0.003s 2022-03-17T21:31:33.7332686Z 2022-03-17T21:31:33.7333082Z OK (skipped=1) 2022-03-17T21:31:33.7333233Z 2022-03-17T21:31:33.7333362Z Generating XML reports... 2022-03-17T21:31:33.7378018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213133.xml 2022-03-17T21:31:34.8742138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:34.8757411Z 2022-03-17T21:31:34.8757917Z Running tests... 2022-03-17T21:31:34.8758455Z ---------------------------------------------------------------------- 2022-03-17T21:31:34.8784126Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:31:34.8784396Z 2022-03-17T21:31:34.8784711Z ---------------------------------------------------------------------- 2022-03-17T21:31:34.8785048Z Ran 1 test in 0.003s 2022-03-17T21:31:34.8785202Z 2022-03-17T21:31:34.8785321Z OK (skipped=1) 2022-03-17T21:31:34.8785452Z 2022-03-17T21:31:34.8785601Z Generating XML reports... 2022-03-17T21:31:34.8830163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213134.xml 2022-03-17T21:31:36.0314447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:36.0329000Z 2022-03-17T21:31:36.0329469Z Running tests... 2022-03-17T21:31:36.0329945Z ---------------------------------------------------------------------- 2022-03-17T21:31:36.0363609Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:31:36.0363900Z 2022-03-17T21:31:36.0364223Z ---------------------------------------------------------------------- 2022-03-17T21:31:36.0364551Z Ran 1 test in 0.004s 2022-03-17T21:31:36.0364719Z 2022-03-17T21:31:36.0364797Z OK (skipped=1) 2022-03-17T21:31:36.0364923Z 2022-03-17T21:31:36.0365044Z Generating XML reports... 2022-03-17T21:31:36.0409657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213136.xml 2022-03-17T21:31:37.1844339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:37.1858743Z 2022-03-17T21:31:37.1859166Z Running tests... 2022-03-17T21:31:37.1859676Z ---------------------------------------------------------------------- 2022-03-17T21:31:37.1886490Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:31:37.1886775Z 2022-03-17T21:31:37.1887046Z ---------------------------------------------------------------------- 2022-03-17T21:31:37.1887393Z Ran 1 test in 0.003s 2022-03-17T21:31:37.1887557Z 2022-03-17T21:31:37.1887668Z OK (skipped=1) 2022-03-17T21:31:37.1887808Z 2022-03-17T21:31:37.1887937Z Generating XML reports... 2022-03-17T21:31:37.1932871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213137.xml 2022-03-17T21:31:38.3474592Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:38.3489115Z 2022-03-17T21:31:38.3489342Z Running tests... 2022-03-17T21:31:38.3489804Z ---------------------------------------------------------------------- 2022-03-17T21:31:38.3513306Z test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:31:38.3513921Z 2022-03-17T21:31:38.3514190Z ---------------------------------------------------------------------- 2022-03-17T21:31:38.3514525Z Ran 1 test in 0.002s 2022-03-17T21:31:38.3514692Z 2022-03-17T21:31:38.3514800Z OK (skipped=1) 2022-03-17T21:31:38.3514935Z 2022-03-17T21:31:38.3515061Z Generating XML reports... 2022-03-17T21:31:38.3559986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213138.xml 2022-03-17T21:31:39.5051313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:39.5065861Z 2022-03-17T21:31:39.5066257Z Running tests... 2022-03-17T21:31:39.5066985Z ---------------------------------------------------------------------- 2022-03-17T21:31:39.5093929Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip (0.003s) 2022-03-17T21:31:39.5094185Z 2022-03-17T21:31:39.5094491Z ---------------------------------------------------------------------- 2022-03-17T21:31:39.5094841Z Ran 1 test in 0.003s 2022-03-17T21:31:39.5095011Z 2022-03-17T21:31:39.5095121Z OK (skipped=1) 2022-03-17T21:31:39.5095259Z 2022-03-17T21:31:39.5095386Z Generating XML reports... 2022-03-17T21:31:39.5139937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213139.xml 2022-03-17T21:31:40.6600108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:40.6614267Z 2022-03-17T21:31:40.6614693Z Running tests... 2022-03-17T21:31:40.6615173Z ---------------------------------------------------------------------- 2022-03-17T21:31:40.6639328Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:31:40.6639649Z 2022-03-17T21:31:40.6639940Z ---------------------------------------------------------------------- 2022-03-17T21:31:40.6640268Z Ran 1 test in 0.003s 2022-03-17T21:31:40.6640435Z 2022-03-17T21:31:40.6640552Z OK (skipped=1) 2022-03-17T21:31:40.6640708Z 2022-03-17T21:31:40.6640817Z Generating XML reports... 2022-03-17T21:31:40.6685952Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213140.xml 2022-03-17T21:31:41.8160475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:41.8174601Z 2022-03-17T21:31:41.8174821Z Running tests... 2022-03-17T21:31:41.8175258Z ---------------------------------------------------------------------- 2022-03-17T21:31:43.3769119Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:31:43.4200403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72007 2022-03-17T21:31:43.4333490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72008 2022-03-17T21:31:44.5263353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:44.5263954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:44.5264669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:44.5265341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:44.5271561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:44.5272073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:44.7380339Z ok (2.920s) 2022-03-17T21:31:44.7380552Z 2022-03-17T21:31:44.7381298Z ---------------------------------------------------------------------- 2022-03-17T21:31:44.7381654Z Ran 1 test in 2.921s 2022-03-17T21:31:44.7382009Z 2022-03-17T21:31:44.7382114Z OK 2022-03-17T21:31:44.7382232Z 2022-03-17T21:31:44.7382365Z Generating XML reports... 2022-03-17T21:31:44.7426805Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213141.xml 2022-03-17T21:31:46.0324808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:46.0338898Z 2022-03-17T21:31:46.0339221Z Running tests... 2022-03-17T21:31:46.0339667Z ---------------------------------------------------------------------- 2022-03-17T21:31:47.5887219Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:31:47.6318971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72122 2022-03-17T21:31:47.6452372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72123 2022-03-17T21:31:48.7502109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:48.7502670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:48.7503431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:48.7504139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:48.7510974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:48.7511532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:48.9501494Z ok (2.916s) 2022-03-17T21:31:48.9501729Z 2022-03-17T21:31:48.9502111Z ---------------------------------------------------------------------- 2022-03-17T21:31:48.9502430Z Ran 1 test in 2.916s 2022-03-17T21:31:48.9502597Z 2022-03-17T21:31:48.9502706Z OK 2022-03-17T21:31:48.9502844Z 2022-03-17T21:31:48.9502981Z Generating XML reports... 2022-03-17T21:31:48.9546485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213146.xml 2022-03-17T21:31:50.2407657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:50.2421310Z 2022-03-17T21:31:50.2421804Z Running tests... 2022-03-17T21:31:50.2422332Z ---------------------------------------------------------------------- 2022-03-17T21:31:50.2444786Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:31:50.2445313Z 2022-03-17T21:31:50.2445640Z ---------------------------------------------------------------------- 2022-03-17T21:31:50.2445990Z Ran 1 test in 0.002s 2022-03-17T21:31:50.2446088Z 2022-03-17T21:31:50.2446178Z OK (skipped=1) 2022-03-17T21:31:50.2446338Z 2022-03-17T21:31:50.2446498Z Generating XML reports... 2022-03-17T21:31:50.2490139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213150.xml 2022-03-17T21:31:51.3965743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:51.3980392Z 2022-03-17T21:31:51.3980807Z Running tests... 2022-03-17T21:31:51.3981302Z ---------------------------------------------------------------------- 2022-03-17T21:31:51.4001887Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:31:51.4002475Z 2022-03-17T21:31:51.4003094Z ---------------------------------------------------------------------- 2022-03-17T21:31:51.4003641Z Ran 1 test in 0.002s 2022-03-17T21:31:51.4003792Z 2022-03-17T21:31:51.4003902Z OK (skipped=1) 2022-03-17T21:31:51.4004061Z 2022-03-17T21:31:51.4004187Z Generating XML reports... 2022-03-17T21:31:51.4048797Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213151.xml 2022-03-17T21:31:52.5469513Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:52.5483508Z 2022-03-17T21:31:52.5484033Z Running tests... 2022-03-17T21:31:52.5484633Z ---------------------------------------------------------------------- 2022-03-17T21:31:54.0949885Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:31:54.1378754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72307 2022-03-17T21:31:54.1507753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72308 2022-03-17T21:31:55.2449792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:31:55.2450365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:31:55.2451143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:55.2451857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:31:55.2559951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:31:55.3463797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:31:57.2587172Z ok (4.710s) 2022-03-17T21:31:57.2587397Z 2022-03-17T21:31:57.2587811Z ---------------------------------------------------------------------- 2022-03-17T21:31:57.2588149Z Ran 1 test in 4.710s 2022-03-17T21:31:57.2588274Z 2022-03-17T21:31:57.2588396Z OK 2022-03-17T21:31:57.2588501Z 2022-03-17T21:31:57.2588636Z Generating XML reports... 2022-03-17T21:31:57.2631465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213152.xml 2022-03-17T21:31:58.5699096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:31:58.5713437Z 2022-03-17T21:31:58.5713744Z Running tests... 2022-03-17T21:31:58.5714194Z ---------------------------------------------------------------------- 2022-03-17T21:32:00.1124174Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:00.1558196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72428 2022-03-17T21:32:00.1688087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72429 2022-03-17T21:32:01.2334080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:01.2335078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:01.2336483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:01.2337473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:01.2443906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:01.3350063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:03.2767055Z ok (4.705s) 2022-03-17T21:32:03.2767344Z 2022-03-17T21:32:03.2767765Z ---------------------------------------------------------------------- 2022-03-17T21:32:03.2768098Z Ran 1 test in 4.705s 2022-03-17T21:32:03.2768244Z 2022-03-17T21:32:03.2768342Z OK 2022-03-17T21:32:03.2768474Z 2022-03-17T21:32:03.2768608Z Generating XML reports... 2022-03-17T21:32:03.2811466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213158.xml 2022-03-17T21:32:04.5732669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:04.5747061Z 2022-03-17T21:32:04.5747306Z Running tests... 2022-03-17T21:32:04.5748215Z ---------------------------------------------------------------------- 2022-03-17T21:32:06.1086197Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:06.1511551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72549 2022-03-17T21:32:06.1638673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72550 2022-03-17T21:32:07.2272205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:07.2272987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:07.2273810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:07.2274489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:07.2381329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:07.3286638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:09.8729385Z ok (5.298s) 2022-03-17T21:32:09.8729748Z 2022-03-17T21:32:09.8730275Z ---------------------------------------------------------------------- 2022-03-17T21:32:09.8730613Z Ran 1 test in 5.298s 2022-03-17T21:32:09.8730782Z 2022-03-17T21:32:09.8730868Z OK 2022-03-17T21:32:09.8731007Z 2022-03-17T21:32:09.8731143Z Generating XML reports... 2022-03-17T21:32:09.8774142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213204.xml 2022-03-17T21:32:11.1628022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:11.1642145Z 2022-03-17T21:32:11.1642578Z Running tests... 2022-03-17T21:32:11.1643059Z ---------------------------------------------------------------------- 2022-03-17T21:32:12.6709908Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:12.7133792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72667 2022-03-17T21:32:12.7262235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72668 2022-03-17T21:32:13.7908667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:13.7909268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:13.7910046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:13.7910738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:13.8017359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:13.8922830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:15.4671653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:32:15.4672242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:32:15.4693875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:32:15.4694389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration. 2022-03-17T21:32:15.9343126Z ok (4.770s) 2022-03-17T21:32:15.9343498Z 2022-03-17T21:32:15.9344094Z ---------------------------------------------------------------------- 2022-03-17T21:32:15.9344710Z Ran 1 test in 4.770s 2022-03-17T21:32:15.9345014Z 2022-03-17T21:32:15.9345535Z OK 2022-03-17T21:32:15.9345755Z 2022-03-17T21:32:15.9346006Z Generating XML reports... 2022-03-17T21:32:15.9389019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213211.xml 2022-03-17T21:32:17.2316116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:17.2330765Z 2022-03-17T21:32:17.2331221Z Running tests... 2022-03-17T21:32:17.2331701Z ---------------------------------------------------------------------- 2022-03-17T21:32:18.7549962Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:18.7979731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72788 2022-03-17T21:32:18.8107553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72789 2022-03-17T21:32:19.9002648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:19.9003196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:19.9003966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:19.9004662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:19.9010939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:19.9011424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:19.9220243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:32:19.9221020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:32:19.9221723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:19.9222539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:20.1154442Z ok (2.882s) 2022-03-17T21:32:20.1154630Z 2022-03-17T21:32:20.1155086Z ---------------------------------------------------------------------- 2022-03-17T21:32:20.1155405Z Ran 1 test in 2.882s 2022-03-17T21:32:20.1155544Z 2022-03-17T21:32:20.1155618Z OK 2022-03-17T21:32:20.1155755Z 2022-03-17T21:32:20.1155889Z Generating XML reports... 2022-03-17T21:32:20.1200439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213217.xml 2022-03-17T21:32:21.4031788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:21.4045727Z 2022-03-17T21:32:21.4046036Z Running tests... 2022-03-17T21:32:21.4046433Z ---------------------------------------------------------------------- 2022-03-17T21:32:22.9542581Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:22.9990139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72913 2022-03-17T21:32:23.0121652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72914 2022-03-17T21:32:24.1256530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:24.1257059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:24.1257871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:24.1258571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:24.1266606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:24.1374258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:24.1374778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:32:24.1375259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:32:24.1375923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:24.1376610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:24.3168319Z ok (2.912s) 2022-03-17T21:32:24.3168556Z 2022-03-17T21:32:24.3168980Z ---------------------------------------------------------------------- 2022-03-17T21:32:24.3169331Z Ran 1 test in 2.912s 2022-03-17T21:32:24.3169472Z 2022-03-17T21:32:24.3169568Z OK 2022-03-17T21:32:24.3169696Z 2022-03-17T21:32:24.3169832Z Generating XML reports... 2022-03-17T21:32:24.3212337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213221.xml 2022-03-17T21:32:25.6139489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:25.6154210Z 2022-03-17T21:32:25.6154841Z Running tests... 2022-03-17T21:32:25.6155355Z ---------------------------------------------------------------------- 2022-03-17T21:32:27.1636681Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:27.2089508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73038 2022-03-17T21:32:27.2225114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73039 2022-03-17T21:32:28.3294011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:28.3294591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:28.3295299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:28.3295999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:28.3302711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:28.3303999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:28.3510312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:32:28.3510819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:32:28.3511425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:28.3512114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:28.5273318Z ok (2.912s) 2022-03-17T21:32:28.5273536Z 2022-03-17T21:32:28.5273902Z ---------------------------------------------------------------------- 2022-03-17T21:32:28.5274238Z Ran 1 test in 2.912s 2022-03-17T21:32:28.5274406Z 2022-03-17T21:32:28.5274507Z OK 2022-03-17T21:32:28.5274625Z 2022-03-17T21:32:28.5274757Z Generating XML reports... 2022-03-17T21:32:28.5316303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213225.xml 2022-03-17T21:32:29.7715381Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:29.7729618Z 2022-03-17T21:32:29.7729870Z Running tests... 2022-03-17T21:32:29.7730316Z ---------------------------------------------------------------------- 2022-03-17T21:32:31.3293296Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:31.3731572Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73163 2022-03-17T21:32:31.3861501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73164 2022-03-17T21:32:32.5009958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:32.5010520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:32.5011493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:32.5012218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:32.5018153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:32.5019210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:32.5126716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:32:32.5127867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:32:32.5128570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:32.5129380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:32:32.6909470Z ok (2.918s) 2022-03-17T21:32:32.6909860Z 2022-03-17T21:32:32.6910578Z ---------------------------------------------------------------------- 2022-03-17T21:32:32.6911143Z Ran 1 test in 2.918s 2022-03-17T21:32:32.6911313Z 2022-03-17T21:32:32.6911385Z OK 2022-03-17T21:32:32.6911534Z 2022-03-17T21:32:32.6911669Z Generating XML reports... 2022-03-17T21:32:32.6953704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213229.xml 2022-03-17T21:32:33.9686346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:33.9700526Z 2022-03-17T21:32:33.9700986Z Running tests... 2022-03-17T21:32:33.9701429Z ---------------------------------------------------------------------- 2022-03-17T21:32:35.5090741Z test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:35.5523003Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73288 2022-03-17T21:32:35.5651201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73289 2022-03-17T21:32:36.6863282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:36.6863813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:36.6864578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:36.6865271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:36.6872128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:36.6872618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:36.8696801Z skip (2.899s) 2022-03-17T21:32:36.8697021Z 2022-03-17T21:32:36.8697410Z ---------------------------------------------------------------------- 2022-03-17T21:32:36.8697762Z Ran 1 test in 2.900s 2022-03-17T21:32:36.8697919Z 2022-03-17T21:32:36.8697997Z OK (skipped=1) 2022-03-17T21:32:36.8698139Z 2022-03-17T21:32:36.8698268Z Generating XML reports... 2022-03-17T21:32:36.8740910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213233.xml 2022-03-17T21:32:38.1535645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:38.1550409Z 2022-03-17T21:32:38.1550845Z Running tests... 2022-03-17T21:32:38.1551322Z ---------------------------------------------------------------------- 2022-03-17T21:32:39.6903992Z test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:39.7334183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73403 2022-03-17T21:32:39.7462793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73404 2022-03-17T21:32:40.8417773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:40.8418305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:40.8419111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:40.8419816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:40.8426389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:40.8427166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:41.0508255Z skip (2.895s) 2022-03-17T21:32:41.0508479Z 2022-03-17T21:32:41.0509119Z ---------------------------------------------------------------------- 2022-03-17T21:32:41.0509525Z Ran 1 test in 2.896s 2022-03-17T21:32:41.0509705Z 2022-03-17T21:32:41.0509745Z OK (skipped=1) 2022-03-17T21:32:41.0509873Z 2022-03-17T21:32:41.0509984Z Generating XML reports... 2022-03-17T21:32:41.0551925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213238.xml 2022-03-17T21:32:42.3401991Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:42.3416358Z 2022-03-17T21:32:42.3416904Z Running tests... 2022-03-17T21:32:42.3417459Z ---------------------------------------------------------------------- 2022-03-17T21:32:43.8868327Z test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:43.9299457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73518 2022-03-17T21:32:43.9431662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73519 2022-03-17T21:32:45.0375911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:45.0376508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:45.0377222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:45.0377918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:45.0484297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:45.1390001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:45.2479844Z skip (2.906s) 2022-03-17T21:32:45.2480044Z 2022-03-17T21:32:45.2480467Z ---------------------------------------------------------------------- 2022-03-17T21:32:45.2480809Z Ran 1 test in 2.906s 2022-03-17T21:32:45.2480966Z 2022-03-17T21:32:45.2481056Z OK (skipped=1) 2022-03-17T21:32:45.2481217Z 2022-03-17T21:32:45.2481346Z Generating XML reports... 2022-03-17T21:32:45.2523915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213242.xml 2022-03-17T21:32:46.5254023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:46.5268341Z 2022-03-17T21:32:46.5268621Z Running tests... 2022-03-17T21:32:46.5269041Z ---------------------------------------------------------------------- 2022-03-17T21:32:48.0670158Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:48.1091022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73633 2022-03-17T21:32:48.1216607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73634 2022-03-17T21:32:49.2453725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:49.2454261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:49.2455004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:49.2455718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:49.2464339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:49.2467216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:49.4260501Z skip (2.899s) 2022-03-17T21:32:49.4261073Z 2022-03-17T21:32:49.4261475Z ---------------------------------------------------------------------- 2022-03-17T21:32:49.4261773Z Ran 1 test in 2.899s 2022-03-17T21:32:49.4261959Z 2022-03-17T21:32:49.4262074Z OK (skipped=1) 2022-03-17T21:32:49.4262240Z 2022-03-17T21:32:49.4262395Z Generating XML reports... 2022-03-17T21:32:49.4306161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213246.xml 2022-03-17T21:32:50.7103819Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:50.7118631Z 2022-03-17T21:32:50.7118807Z Running tests... 2022-03-17T21:32:50.7119599Z ---------------------------------------------------------------------- 2022-03-17T21:32:52.2499895Z test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:52.2926122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73748 2022-03-17T21:32:52.3055539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73749 2022-03-17T21:32:53.3934435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:53.3934995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:53.3935792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:53.3936490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:53.4044211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:53.4949151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:53.7104272Z ok (2.998s) 2022-03-17T21:32:53.7104697Z 2022-03-17T21:32:53.7105092Z ---------------------------------------------------------------------- 2022-03-17T21:32:53.7105407Z Ran 1 test in 2.999s 2022-03-17T21:32:53.7105524Z 2022-03-17T21:32:53.7105576Z OK 2022-03-17T21:32:53.7105712Z 2022-03-17T21:32:53.7105847Z Generating XML reports... 2022-03-17T21:32:53.7147915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213250.xml 2022-03-17T21:32:54.9937454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:54.9952571Z 2022-03-17T21:32:54.9953057Z Running tests... 2022-03-17T21:32:54.9953541Z ---------------------------------------------------------------------- 2022-03-17T21:32:56.5198416Z test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:32:56.5631971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73867 2022-03-17T21:32:56.5761910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73868 2022-03-17T21:32:57.6384570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:32:57.6385413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:32:57.6386123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:57.6386818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:32:57.6393330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:32:57.6393865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:32:57.8806760Z ok (2.885s) 2022-03-17T21:32:57.8806942Z 2022-03-17T21:32:57.8807360Z ---------------------------------------------------------------------- 2022-03-17T21:32:57.8807717Z Ran 1 test in 2.885s 2022-03-17T21:32:57.8807858Z 2022-03-17T21:32:57.8807949Z OK 2022-03-17T21:32:57.8808010Z 2022-03-17T21:32:57.8808119Z Generating XML reports... 2022-03-17T21:32:57.8850882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213254.xml 2022-03-17T21:32:59.1667664Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:32:59.1682219Z 2022-03-17T21:32:59.1682623Z Running tests... 2022-03-17T21:32:59.1683133Z ---------------------------------------------------------------------- 2022-03-17T21:32:59.1707000Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:32:59.1707341Z 2022-03-17T21:32:59.1707634Z ---------------------------------------------------------------------- 2022-03-17T21:32:59.1707954Z Ran 1 test in 0.003s 2022-03-17T21:32:59.1708135Z 2022-03-17T21:32:59.1708234Z OK (skipped=1) 2022-03-17T21:32:59.1708337Z 2022-03-17T21:32:59.1708452Z Generating XML reports... 2022-03-17T21:32:59.1754434Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213259.xml 2022-03-17T21:33:00.3145465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:00.3160293Z 2022-03-17T21:33:00.3160667Z Running tests... 2022-03-17T21:33:00.3161180Z ---------------------------------------------------------------------- 2022-03-17T21:33:01.8752764Z test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:01.9197467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74021 2022-03-17T21:33:01.9332378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74022 2022-03-17T21:33:03.0135229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:03.0135772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:03.0136481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:03.0137162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:03.0243451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:03.1147911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:03.3380173Z ok (3.022s) 2022-03-17T21:33:03.3380754Z 2022-03-17T21:33:03.3381163Z ---------------------------------------------------------------------- 2022-03-17T21:33:03.3381445Z Ran 1 test in 3.022s 2022-03-17T21:33:03.3381574Z 2022-03-17T21:33:03.3381667Z OK 2022-03-17T21:33:03.3381836Z 2022-03-17T21:33:03.3382012Z Generating XML reports... 2022-03-17T21:33:03.3424142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213300.xml 2022-03-17T21:33:04.6204053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:04.6218785Z 2022-03-17T21:33:04.6218959Z Running tests... 2022-03-17T21:33:04.6219418Z ---------------------------------------------------------------------- 2022-03-17T21:33:06.1683931Z test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:06.2116076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74140 2022-03-17T21:33:06.2243940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74141 2022-03-17T21:33:07.3510160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:07.3510699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:07.3511474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:07.3512204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:07.3518843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:07.3519323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:07.5288320Z ok (2.907s) 2022-03-17T21:33:07.5288495Z 2022-03-17T21:33:07.5288885Z ---------------------------------------------------------------------- 2022-03-17T21:33:07.5289199Z Ran 1 test in 2.907s 2022-03-17T21:33:07.5289377Z 2022-03-17T21:33:07.5289474Z OK 2022-03-17T21:33:07.5289603Z 2022-03-17T21:33:07.5289714Z Generating XML reports... 2022-03-17T21:33:07.5332170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213304.xml 2022-03-17T21:33:08.8058802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:08.8073339Z 2022-03-17T21:33:08.8073595Z Running tests... 2022-03-17T21:33:08.8074034Z ---------------------------------------------------------------------- 2022-03-17T21:33:08.8097991Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:08.8098254Z 2022-03-17T21:33:08.8098520Z ---------------------------------------------------------------------- 2022-03-17T21:33:08.8098871Z Ran 1 test in 0.003s 2022-03-17T21:33:08.8099036Z 2022-03-17T21:33:08.8099159Z OK (skipped=1) 2022-03-17T21:33:08.8099296Z 2022-03-17T21:33:08.8099418Z Generating XML reports... 2022-03-17T21:33:08.8144108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213308.xml 2022-03-17T21:33:09.9595391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:09.9609339Z 2022-03-17T21:33:09.9609490Z Running tests... 2022-03-17T21:33:09.9610193Z ---------------------------------------------------------------------- 2022-03-17T21:33:09.9634700Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:09.9634972Z 2022-03-17T21:33:09.9635354Z ---------------------------------------------------------------------- 2022-03-17T21:33:09.9635707Z Ran 1 test in 0.003s 2022-03-17T21:33:09.9635869Z 2022-03-17T21:33:09.9636265Z OK (skipped=1) 2022-03-17T21:33:09.9636425Z 2022-03-17T21:33:09.9636533Z Generating XML reports... 2022-03-17T21:33:09.9680987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213309.xml 2022-03-17T21:33:11.1061497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:11.1075998Z 2022-03-17T21:33:11.1076297Z Running tests... 2022-03-17T21:33:11.1076737Z ---------------------------------------------------------------------- 2022-03-17T21:33:12.6664450Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:12.7095609Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74329 2022-03-17T21:33:12.7223405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74330 2022-03-17T21:33:13.8447288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:13.8447868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:13.8448668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:13.8449344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:13.8455396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:13.8456225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:14.0270716Z ok (2.919s) 2022-03-17T21:33:14.0270960Z 2022-03-17T21:33:14.0271573Z ---------------------------------------------------------------------- 2022-03-17T21:33:14.0271896Z Ran 1 test in 2.919s 2022-03-17T21:33:14.0272070Z 2022-03-17T21:33:14.0273183Z OK 2022-03-17T21:33:14.0273524Z 2022-03-17T21:33:14.0273787Z Generating XML reports... 2022-03-17T21:33:14.0313891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213311.xml 2022-03-17T21:33:15.3109678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:15.3123813Z 2022-03-17T21:33:15.3124313Z Running tests... 2022-03-17T21:33:15.3124793Z ---------------------------------------------------------------------- 2022-03-17T21:33:16.8425961Z test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:16.8851439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74448 2022-03-17T21:33:16.8978955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74449 2022-03-17T21:33:17.9982831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:17.9983389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:17.9984162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:17.9984861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:17.9990592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:17.9991036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:18.2027194Z ok (2.890s) 2022-03-17T21:33:18.2027437Z 2022-03-17T21:33:18.2027819Z ---------------------------------------------------------------------- 2022-03-17T21:33:18.2028140Z Ran 1 test in 2.890s 2022-03-17T21:33:18.2028292Z 2022-03-17T21:33:18.2028393Z OK 2022-03-17T21:33:18.2028486Z 2022-03-17T21:33:18.2028600Z Generating XML reports... 2022-03-17T21:33:18.2070970Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213315.xml 2022-03-17T21:33:19.4838647Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:19.4856861Z 2022-03-17T21:33:19.4857106Z Running tests... 2022-03-17T21:33:19.4857550Z ---------------------------------------------------------------------- 2022-03-17T21:33:21.0216694Z test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:21.0649479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74567 2022-03-17T21:33:21.0782071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74568 2022-03-17T21:33:22.2099525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:22.2100080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:22.2101175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:22.2101857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:22.2107432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:22.2108457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:22.3829465Z ok (2.897s) 2022-03-17T21:33:22.3829713Z 2022-03-17T21:33:22.3830099Z ---------------------------------------------------------------------- 2022-03-17T21:33:22.3830438Z Ran 1 test in 2.897s 2022-03-17T21:33:22.3830623Z 2022-03-17T21:33:22.3830697Z OK 2022-03-17T21:33:22.3830842Z 2022-03-17T21:33:22.3830972Z Generating XML reports... 2022-03-17T21:33:22.3872508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213319.xml 2022-03-17T21:33:23.6646620Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:23.6661202Z 2022-03-17T21:33:23.6661509Z Running tests... 2022-03-17T21:33:23.6662199Z ---------------------------------------------------------------------- 2022-03-17T21:33:25.2113596Z test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:25.2544361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74682 2022-03-17T21:33:25.2672540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74683 2022-03-17T21:33:26.3637812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:26.3638344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:26.3639099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:26.3639810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:26.3746619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:26.4649730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:26.6721431Z ok (3.006s) 2022-03-17T21:33:26.6721654Z 2022-03-17T21:33:26.6722090Z ---------------------------------------------------------------------- 2022-03-17T21:33:26.6722372Z Ran 1 test in 3.006s 2022-03-17T21:33:26.6722542Z 2022-03-17T21:33:26.6722645Z OK 2022-03-17T21:33:26.6722804Z 2022-03-17T21:33:26.6722925Z Generating XML reports... 2022-03-17T21:33:26.6766594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213323.xml 2022-03-17T21:33:27.9568976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:27.9583503Z 2022-03-17T21:33:27.9583756Z Running tests... 2022-03-17T21:33:27.9584184Z ---------------------------------------------------------------------- 2022-03-17T21:33:27.9604701Z test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:27.9605075Z 2022-03-17T21:33:27.9605366Z ---------------------------------------------------------------------- 2022-03-17T21:33:27.9605692Z Ran 1 test in 0.002s 2022-03-17T21:33:27.9605859Z 2022-03-17T21:33:27.9605972Z OK (skipped=1) 2022-03-17T21:33:27.9606182Z 2022-03-17T21:33:27.9606276Z Generating XML reports... 2022-03-17T21:33:27.9651134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213327.xml 2022-03-17T21:33:29.1162748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:29.1177485Z 2022-03-17T21:33:29.1177740Z Running tests... 2022-03-17T21:33:29.1178206Z ---------------------------------------------------------------------- 2022-03-17T21:33:29.1199860Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:33:29.1200357Z 2022-03-17T21:33:29.1200650Z ---------------------------------------------------------------------- 2022-03-17T21:33:29.1200987Z Ran 1 test in 0.002s 2022-03-17T21:33:29.1201132Z 2022-03-17T21:33:29.1201243Z OK (skipped=1) 2022-03-17T21:33:29.1201404Z 2022-03-17T21:33:29.1201532Z Generating XML reports... 2022-03-17T21:33:29.1247017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213329.xml 2022-03-17T21:33:30.2675446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:30.2689482Z 2022-03-17T21:33:30.2689730Z Running tests... 2022-03-17T21:33:30.2690162Z ---------------------------------------------------------------------- 2022-03-17T21:33:31.8105605Z test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:31.8534921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74871 2022-03-17T21:33:31.8663301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74872 2022-03-17T21:33:32.9517285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:32.9517827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:32.9518595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:32.9519276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:32.9526175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:32.9526725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:32.9634273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:33:32.9634727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:33:32.9635387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:33:32.9636238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:33:33.1710265Z ok (2.902s) 2022-03-17T21:33:33.1710691Z 2022-03-17T21:33:33.1711227Z ---------------------------------------------------------------------- 2022-03-17T21:33:33.1711552Z Ran 1 test in 2.902s 2022-03-17T21:33:33.1711703Z 2022-03-17T21:33:33.1711785Z OK 2022-03-17T21:33:33.1712021Z 2022-03-17T21:33:33.1712069Z Generating XML reports... 2022-03-17T21:33:33.1754217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213330.xml 2022-03-17T21:33:34.4503930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:34.4518491Z 2022-03-17T21:33:34.4518909Z Running tests... 2022-03-17T21:33:34.4519439Z ---------------------------------------------------------------------- 2022-03-17T21:33:35.9946002Z test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:36.0376905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74996 2022-03-17T21:33:36.0506215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74997 2022-03-17T21:33:37.1812629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:37.1813161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:37.1814003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:37.1814642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:37.1821358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:37.1822576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:37.3551789Z skip (2.903s) 2022-03-17T21:33:37.3552207Z 2022-03-17T21:33:37.3552940Z ---------------------------------------------------------------------- 2022-03-17T21:33:37.3553500Z Ran 1 test in 2.903s 2022-03-17T21:33:37.3553814Z 2022-03-17T21:33:37.3554046Z OK (skipped=1) 2022-03-17T21:33:37.3554288Z 2022-03-17T21:33:37.3554503Z Generating XML reports... 2022-03-17T21:33:37.3598693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213334.xml 2022-03-17T21:33:38.6437214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:38.6450976Z 2022-03-17T21:33:38.6451391Z Running tests... 2022-03-17T21:33:38.6451891Z ---------------------------------------------------------------------- 2022-03-17T21:33:40.1831762Z test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:40.2262911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75111 2022-03-17T21:33:40.2391967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75112 2022-03-17T21:33:41.3348652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:41.3349250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:41.3350052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:41.3350726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:41.3457283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:41.4362206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:41.6442244Z ok (2.999s) 2022-03-17T21:33:41.6442447Z 2022-03-17T21:33:41.6443092Z ---------------------------------------------------------------------- 2022-03-17T21:33:41.6443636Z Ran 1 test in 2.999s 2022-03-17T21:33:41.6443824Z 2022-03-17T21:33:41.6443904Z OK 2022-03-17T21:33:41.6444042Z 2022-03-17T21:33:41.6444170Z Generating XML reports... 2022-03-17T21:33:41.6487535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213338.xml 2022-03-17T21:33:42.9255322Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:42.9270047Z 2022-03-17T21:33:42.9270449Z Running tests... 2022-03-17T21:33:42.9270961Z ---------------------------------------------------------------------- 2022-03-17T21:33:44.5002888Z test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:44.5439550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75226 2022-03-17T21:33:44.5569414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75227 2022-03-17T21:33:45.6181412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:45.6181969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:45.6182749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:45.6183460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:45.6292162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:45.7194472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:45.8615628Z ok (2.934s) 2022-03-17T21:33:45.8615782Z 2022-03-17T21:33:45.8616143Z ---------------------------------------------------------------------- 2022-03-17T21:33:45.8616494Z Ran 1 test in 2.935s 2022-03-17T21:33:45.8616671Z 2022-03-17T21:33:45.8616744Z OK 2022-03-17T21:33:45.8616888Z 2022-03-17T21:33:45.8617001Z Generating XML reports... 2022-03-17T21:33:45.8660055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213342.xml 2022-03-17T21:33:47.1435255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:47.1449487Z 2022-03-17T21:33:47.1449886Z Running tests... 2022-03-17T21:33:47.1450393Z ---------------------------------------------------------------------- 2022-03-17T21:33:48.6839415Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:48.7272167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75341 2022-03-17T21:33:48.7401993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75342 2022-03-17T21:33:49.8040525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:49.8041040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:49.8041812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:49.8042520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:49.8149909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:49.9051632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:50.0453433Z ok (2.900s) 2022-03-17T21:33:50.0453769Z 2022-03-17T21:33:50.0454196Z ---------------------------------------------------------------------- 2022-03-17T21:33:50.0454528Z Ran 1 test in 2.900s 2022-03-17T21:33:50.0454665Z 2022-03-17T21:33:50.0454734Z OK 2022-03-17T21:33:50.0454853Z 2022-03-17T21:33:50.0454965Z Generating XML reports... 2022-03-17T21:33:50.0497688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213347.xml 2022-03-17T21:33:51.3236316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:51.3250048Z 2022-03-17T21:33:51.3250471Z Running tests... 2022-03-17T21:33:51.3251409Z ---------------------------------------------------------------------- 2022-03-17T21:33:52.8404304Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:52.8831892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75456 2022-03-17T21:33:52.8961128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75457 2022-03-17T21:33:53.9822331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:53.9823114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:53.9823911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:53.9824587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:53.9931871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:54.0835355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:54.3010819Z ok (2.976s) 2022-03-17T21:33:54.3011097Z 2022-03-17T21:33:54.3011503Z ---------------------------------------------------------------------- 2022-03-17T21:33:54.3011840Z Ran 1 test in 2.976s 2022-03-17T21:33:54.3012010Z 2022-03-17T21:33:54.3012064Z OK 2022-03-17T21:33:54.3012151Z 2022-03-17T21:33:54.3012360Z Generating XML reports... 2022-03-17T21:33:54.3055164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213351.xml 2022-03-17T21:33:55.5487148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:55.5501158Z 2022-03-17T21:33:55.5501621Z Running tests... 2022-03-17T21:33:55.5502122Z ---------------------------------------------------------------------- 2022-03-17T21:33:57.0978421Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:33:57.1406452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75575 2022-03-17T21:33:57.1533358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75576 2022-03-17T21:33:58.2337089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:33:58.2337597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:33:58.2338382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:58.2339077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:33:58.2445605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:33:58.3347991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:33:58.5585201Z ok (3.008s) 2022-03-17T21:33:58.5585620Z 2022-03-17T21:33:58.5586038Z ---------------------------------------------------------------------- 2022-03-17T21:33:58.5586323Z Ran 1 test in 3.008s 2022-03-17T21:33:58.5586459Z 2022-03-17T21:33:58.5586546Z OK 2022-03-17T21:33:58.5586683Z 2022-03-17T21:33:58.5586795Z Generating XML reports... 2022-03-17T21:33:58.5630235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213355.xml 2022-03-17T21:33:59.8546384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:33:59.8560566Z 2022-03-17T21:33:59.8560768Z Running tests... 2022-03-17T21:33:59.8561473Z ---------------------------------------------------------------------- 2022-03-17T21:34:01.4112437Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:01.4573197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75694 2022-03-17T21:34:01.4716445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75695 2022-03-17T21:34:02.5463235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:02.5463776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:02.5464703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:02.5465415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:02.5571837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:02.6475118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:02.8765249Z ok (3.020s) 2022-03-17T21:34:02.8765479Z 2022-03-17T21:34:02.8765856Z ---------------------------------------------------------------------- 2022-03-17T21:34:02.8766194Z Ran 1 test in 3.020s 2022-03-17T21:34:02.8766359Z 2022-03-17T21:34:02.8766458Z OK 2022-03-17T21:34:02.8766575Z 2022-03-17T21:34:02.8766708Z Generating XML reports... 2022-03-17T21:34:02.8809353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213359.xml 2022-03-17T21:34:04.1594388Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:04.1608601Z 2022-03-17T21:34:04.1608903Z Running tests... 2022-03-17T21:34:04.1609383Z ---------------------------------------------------------------------- 2022-03-17T21:34:04.1629416Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:34:04.1629687Z 2022-03-17T21:34:04.1641657Z ---------------------------------------------------------------------- 2022-03-17T21:34:04.1642063Z Ran 1 test in 0.002s 2022-03-17T21:34:04.1642197Z 2022-03-17T21:34:04.1642292Z OK (skipped=1) 2022-03-17T21:34:04.1642444Z 2022-03-17T21:34:04.1642573Z Generating XML reports... 2022-03-17T21:34:04.1674890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213404.xml 2022-03-17T21:34:05.2781555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:05.2795628Z 2022-03-17T21:34:05.2795767Z Running tests... 2022-03-17T21:34:05.2796268Z ---------------------------------------------------------------------- 2022-03-17T21:34:05.2817005Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:34:05.2817305Z 2022-03-17T21:34:05.2817596Z ---------------------------------------------------------------------- 2022-03-17T21:34:05.2817926Z Ran 1 test in 0.002s 2022-03-17T21:34:05.2818092Z 2022-03-17T21:34:05.2818183Z OK (skipped=1) 2022-03-17T21:34:05.2818281Z 2022-03-17T21:34:05.2818407Z Generating XML reports... 2022-03-17T21:34:05.2862906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213405.xml 2022-03-17T21:34:06.4314749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:06.4328990Z 2022-03-17T21:34:06.4329511Z Running tests... 2022-03-17T21:34:06.4330008Z ---------------------------------------------------------------------- 2022-03-17T21:34:06.4353140Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip (0.002s) 2022-03-17T21:34:06.4353662Z 2022-03-17T21:34:06.4353893Z ---------------------------------------------------------------------- 2022-03-17T21:34:06.4354508Z Ran 1 test in 0.002s 2022-03-17T21:34:06.4354653Z 2022-03-17T21:34:06.4354763Z OK (skipped=1) 2022-03-17T21:34:06.4354917Z 2022-03-17T21:34:06.4355040Z Generating XML reports... 2022-03-17T21:34:06.4398875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213406.xml 2022-03-17T21:34:07.5826088Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:07.5839883Z 2022-03-17T21:34:07.5840280Z Running tests... 2022-03-17T21:34:07.5840777Z ---------------------------------------------------------------------- 2022-03-17T21:34:09.1612430Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:09.2043394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75918 2022-03-17T21:34:09.2171745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75919 2022-03-17T21:34:10.3236012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:10.3236613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:10.3237336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:10.3238012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:10.3347331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:10.4249213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:10.6219056Z ok (3.038s) 2022-03-17T21:34:10.6219207Z 2022-03-17T21:34:10.6219658Z ---------------------------------------------------------------------- 2022-03-17T21:34:10.6219993Z Ran 1 test in 3.038s 2022-03-17T21:34:10.6220153Z 2022-03-17T21:34:10.6220258Z OK 2022-03-17T21:34:10.6220557Z 2022-03-17T21:34:10.6220690Z Generating XML reports... 2022-03-17T21:34:10.6262684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213407.xml 2022-03-17T21:34:11.9026452Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:11.9040683Z 2022-03-17T21:34:11.9041146Z Running tests... 2022-03-17T21:34:11.9041597Z ---------------------------------------------------------------------- 2022-03-17T21:34:13.4458764Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:13.4888216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76037 2022-03-17T21:34:13.5016351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76038 2022-03-17T21:34:14.5672077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:14.5672654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:14.5673381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:14.5674071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:14.5781668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:14.6683930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:14.8062115Z ok (2.902s) 2022-03-17T21:34:14.8062540Z 2022-03-17T21:34:14.8063248Z ---------------------------------------------------------------------- 2022-03-17T21:34:14.8063603Z Ran 1 test in 2.902s 2022-03-17T21:34:14.8063771Z 2022-03-17T21:34:14.8063863Z OK 2022-03-17T21:34:14.8064288Z 2022-03-17T21:34:14.8064402Z Generating XML reports... 2022-03-17T21:34:14.8108025Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213411.xml 2022-03-17T21:34:16.0915720Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:16.0929973Z 2022-03-17T21:34:16.0930206Z Running tests... 2022-03-17T21:34:16.0930671Z ---------------------------------------------------------------------- 2022-03-17T21:34:17.6535612Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:17.6965993Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76152 2022-03-17T21:34:17.7094385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76153 2022-03-17T21:34:18.8091187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:18.8091777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:18.8092534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:18.8093212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:18.8201184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:18.9101987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:19.1141419Z ok (3.021s) 2022-03-17T21:34:19.1141737Z 2022-03-17T21:34:19.1142181Z ---------------------------------------------------------------------- 2022-03-17T21:34:19.1142519Z Ran 1 test in 3.021s 2022-03-17T21:34:19.1142691Z 2022-03-17T21:34:19.1142784Z OK 2022-03-17T21:34:19.1142901Z 2022-03-17T21:34:19.1143028Z Generating XML reports... 2022-03-17T21:34:19.1186452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213416.xml 2022-03-17T21:34:20.3643040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:20.3657096Z 2022-03-17T21:34:20.3657355Z Running tests... 2022-03-17T21:34:20.3657787Z ---------------------------------------------------------------------- 2022-03-17T21:34:21.9077722Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:21.9506346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76271 2022-03-17T21:34:21.9633104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76272 2022-03-17T21:34:23.0146346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:23.0147372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:23.0148808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:23.0150166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:23.0155298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:23.0156049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:23.1678767Z ok (2.802s) 2022-03-17T21:34:23.1679012Z 2022-03-17T21:34:23.1679420Z ---------------------------------------------------------------------- 2022-03-17T21:34:23.1679740Z Ran 1 test in 2.802s 2022-03-17T21:34:23.1679907Z 2022-03-17T21:34:23.1679975Z OK 2022-03-17T21:34:23.1680115Z 2022-03-17T21:34:23.1680233Z Generating XML reports... 2022-03-17T21:34:23.1723422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213420.xml 2022-03-17T21:34:24.4607940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:24.4624024Z 2022-03-17T21:34:24.4624521Z Running tests... 2022-03-17T21:34:24.4625015Z ---------------------------------------------------------------------- 2022-03-17T21:34:26.0165760Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:26.0598339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76390 2022-03-17T21:34:26.0730787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76391 2022-03-17T21:34:27.1743367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:27.1743950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:27.1744773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:27.1745451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:27.1752637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:27.1753711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:27.3780983Z ok (2.915s) 2022-03-17T21:34:27.3781582Z 2022-03-17T21:34:27.3781997Z ---------------------------------------------------------------------- 2022-03-17T21:34:27.3782273Z Ran 1 test in 2.916s 2022-03-17T21:34:27.3782417Z 2022-03-17T21:34:27.3782529Z OK 2022-03-17T21:34:27.3782647Z 2022-03-17T21:34:27.3782786Z Generating XML reports... 2022-03-17T21:34:27.3826767Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213424.xml 2022-03-17T21:34:28.6606048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:28.6621613Z 2022-03-17T21:34:28.6621850Z Running tests... 2022-03-17T21:34:28.6622268Z ---------------------------------------------------------------------- 2022-03-17T21:34:30.1988400Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:30.2416292Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76595 2022-03-17T21:34:30.2545440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76596 2022-03-17T21:34:31.3593493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:31.3594078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:31.3594805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:31.3595472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:31.3603706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:31.3606179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:32.9619240Z ok (4.299s) 2022-03-17T21:34:32.9619464Z 2022-03-17T21:34:32.9619854Z ---------------------------------------------------------------------- 2022-03-17T21:34:32.9620171Z Ran 1 test in 4.300s 2022-03-17T21:34:32.9620579Z 2022-03-17T21:34:32.9620663Z OK 2022-03-17T21:34:32.9622928Z 2022-03-17T21:34:32.9623406Z Generating XML reports... 2022-03-17T21:34:32.9663530Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213428.xml 2022-03-17T21:34:34.2632346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:34.2646600Z 2022-03-17T21:34:34.2646840Z Running tests... 2022-03-17T21:34:34.2647536Z ---------------------------------------------------------------------- 2022-03-17T21:34:35.8253577Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:35.8682366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76802 2022-03-17T21:34:35.8812407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76803 2022-03-17T21:34:36.9381529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:36.9382103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:36.9382886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:36.9383570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:36.9491507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:37.0395817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:38.8887436Z ok (4.624s) 2022-03-17T21:34:38.8887753Z 2022-03-17T21:34:38.8888184Z ---------------------------------------------------------------------- 2022-03-17T21:34:38.8888516Z Ran 1 test in 4.624s 2022-03-17T21:34:38.8888665Z 2022-03-17T21:34:38.8888740Z OK 2022-03-17T21:34:38.8888875Z 2022-03-17T21:34:38.8889021Z Generating XML reports... 2022-03-17T21:34:38.8932193Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213434.xml 2022-03-17T21:34:40.1970684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:40.1985226Z 2022-03-17T21:34:40.1985623Z Running tests... 2022-03-17T21:34:40.1986139Z ---------------------------------------------------------------------- 2022-03-17T21:34:41.7447787Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:41.7877953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76923 2022-03-17T21:34:41.8008589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76924 2022-03-17T21:34:42.8987738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:42.8988306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:42.8989064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:42.8989755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:42.9092896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:42.9995508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:43.2057336Z ok (3.007s) 2022-03-17T21:34:43.2057590Z 2022-03-17T21:34:43.2057983Z ---------------------------------------------------------------------- 2022-03-17T21:34:43.2058300Z Ran 1 test in 3.007s 2022-03-17T21:34:43.2058471Z 2022-03-17T21:34:43.2058540Z OK 2022-03-17T21:34:43.2058670Z 2022-03-17T21:34:43.2058801Z Generating XML reports... 2022-03-17T21:34:43.2100859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213440.xml 2022-03-17T21:34:44.4930818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:44.4945445Z 2022-03-17T21:34:44.4945915Z Running tests... 2022-03-17T21:34:44.4946364Z ---------------------------------------------------------------------- 2022-03-17T21:34:46.0370290Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:46.0799887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77042 2022-03-17T21:34:46.0928539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77043 2022-03-17T21:34:47.1875124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:47.1876149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:47.1877472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:47.1878194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:47.1982914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:47.2890540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:48.9014883Z ok (4.407s) 2022-03-17T21:34:48.9015228Z 2022-03-17T21:34:48.9015544Z ---------------------------------------------------------------------- 2022-03-17T21:34:48.9015886Z Ran 1 test in 4.407s 2022-03-17T21:34:48.9016052Z 2022-03-17T21:34:48.9016129Z OK 2022-03-17T21:34:48.9016266Z 2022-03-17T21:34:48.9016399Z Generating XML reports... 2022-03-17T21:34:48.9059095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213444.xml 2022-03-17T21:34:50.2112732Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:50.2127465Z 2022-03-17T21:34:50.2127659Z Running tests... 2022-03-17T21:34:50.2128117Z ---------------------------------------------------------------------- 2022-03-17T21:34:51.7508472Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:51.7941392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77159 2022-03-17T21:34:51.8068405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77160 2022-03-17T21:34:52.8891064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:52.8891634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:52.8892369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:52.8893043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:52.8999444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:52.9905984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:54.5425701Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:34:54.5454637Z [W reducer.cpp:1263] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) 2022-03-17T21:34:54.8144023Z ok (4.601s) 2022-03-17T21:34:54.8144218Z 2022-03-17T21:34:54.8144702Z ---------------------------------------------------------------------- 2022-03-17T21:34:54.8145150Z Ran 1 test in 4.602s 2022-03-17T21:34:54.8145320Z 2022-03-17T21:34:54.8145394Z OK 2022-03-17T21:34:54.8145535Z 2022-03-17T21:34:54.8145667Z Generating XML reports... 2022-03-17T21:34:54.8187530Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213450.xml 2022-03-17T21:34:56.1026697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:34:56.1040640Z 2022-03-17T21:34:56.1040786Z Running tests... 2022-03-17T21:34:56.1041669Z ---------------------------------------------------------------------- 2022-03-17T21:34:57.6370887Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:34:57.6799929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77280 2022-03-17T21:34:57.6932839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77281 2022-03-17T21:34:58.7617171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:34:58.7617718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:34:58.7618531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:58.7619225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:34:58.7725300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:34:58.8628058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:34:58.8835400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:34:58.8835970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:34:58.8836696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:58.8837379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:34:58.9043910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:34:58.9044942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:34:58.9045706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:34:58.9046400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:35:05.6083776Z ok (9.504s) 2022-03-17T21:35:05.6084157Z 2022-03-17T21:35:05.6084919Z ---------------------------------------------------------------------- 2022-03-17T21:35:05.6085532Z Ran 1 test in 9.504s 2022-03-17T21:35:05.6085703Z 2022-03-17T21:35:05.6085795Z OK 2022-03-17T21:35:05.6085930Z 2022-03-17T21:35:05.6086062Z Generating XML reports... 2022-03-17T21:35:05.6127722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213456.xml 2022-03-17T21:35:06.9167458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn 2022-03-17T21:35:06.9182687Z 2022-03-17T21:35:06.9183140Z Running tests... 2022-03-17T21:35:06.9183595Z ---------------------------------------------------------------------- 2022-03-17T21:35:08.4606503Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:08.5040235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77409 2022-03-17T21:35:08.5168395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77410 2022-03-17T21:35:09.6105790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1 2022-03-17T21:35:09.6106513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:35:09.6107297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:09.6107986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. 2022-03-17T21:35:09.6213974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1 2022-03-17T21:35:09.7118200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0 2022-03-17T21:35:09.7228568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1 2022-03-17T21:35:09.7229086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0 2022-03-17T21:35:09.7229756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:35:09.7230446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes. 2022-03-17T21:35:09.7439198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1 2022-03-17T21:35:09.7439696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0 2022-03-17T21:35:09.7440380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:35:09.7441047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes. 2022-03-17T21:35:16.4327971Z ok (9.514s) 2022-03-17T21:35:16.4328344Z 2022-03-17T21:35:16.4329095Z ---------------------------------------------------------------------- 2022-03-17T21:35:16.4329680Z Ran 1 test in 9.515s 2022-03-17T21:35:16.4329829Z 2022-03-17T21:35:16.4329931Z OK 2022-03-17T21:35:16.4330065Z 2022-03-17T21:35:16.4330218Z Generating XML reports... 2022-03-17T21:35:16.4372179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213506.xml 2022-03-17T21:35:16.7919421Z Running distributed/test_launcher ... [2022-03-17 21:35:16.791416] 2022-03-17T21:35:17.8874894Z Executing ['/opt/conda/bin/python', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 21:35:16.791506] 2022-03-17T21:35:17.8875547Z Test results will be stored in test-reports/dist-gloo/distributed.test_launcher 2022-03-17T21:35:17.8890433Z 2022-03-17T21:35:17.8890758Z Running tests... 2022-03-17T21:35:17.8891684Z ---------------------------------------------------------------------- 2022-03-17T21:35:19.4496969Z test_launch_user_script (__main__.TestDistributedLaunch) ... /opt/conda/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated 2022-03-17T21:35:19.4497762Z and will be removed in future. Use torchrun. 2022-03-17T21:35:19.4498674Z Note that --use_env is set by default in torchrun. 2022-03-17T21:35:19.4499577Z If your script expects `--local_rank` argument to be set, please 2022-03-17T21:35:19.4500897Z change it to read from `os.environ['LOCAL_RANK']` instead. See 2022-03-17T21:35:19.4502038Z https://pytorch.org/docs/stable/distributed.html#launch-utility for 2022-03-17T21:35:19.4502394Z further instructions 2022-03-17T21:35:19.4502572Z 2022-03-17T21:35:19.4502682Z FutureWarning, 2022-03-17T21:35:19.4512568Z WARNING:torch.distributed.run: 2022-03-17T21:35:19.4513213Z ***************************************** 2022-03-17T21:35:19.4514087Z Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 2022-03-17T21:35:19.4514598Z ***************************************** 2022-03-17T21:35:19.5006138Z Success, smoke test 2022-03-17T21:35:19.5230628Z Success, smoke test 2022-03-17T21:35:19.5466003Z Success, smoke test 2022-03-17T21:35:19.5703218Z Success, smoke test 2022-03-17T21:35:20.5560593Z ok (2.667s) 2022-03-17T21:35:20.5562507Z 2022-03-17T21:35:20.5563272Z ---------------------------------------------------------------------- 2022-03-17T21:35:20.5564004Z Ran 1 test in 2.667s 2022-03-17T21:35:20.5564257Z 2022-03-17T21:35:20.5564387Z OK 2022-03-17T21:35:20.5564634Z 2022-03-17T21:35:20.5564757Z Generating XML reports... 2022-03-17T21:35:20.5612519Z Generated XML report: test-reports/dist-gloo/distributed.test_launcher/TEST-TestDistributedLaunch-20220317213517.xml 2022-03-17T21:35:20.8018528Z Running distributed/test_nccl ... [2022-03-17 21:35:20.801364] 2022-03-17T21:35:20.8019521Z Executing ['/opt/conda/bin/python', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 21:35:20.801468] 2022-03-17T21:35:23.1743175Z Test results will be stored in test-reports/dist-gloo/distributed.test_nccl 2022-03-17T21:35:23.1766351Z 2022-03-17T21:35:23.1766667Z Running tests... 2022-03-17T21:35:23.1767546Z ---------------------------------------------------------------------- 2022-03-17T21:35:24.2682672Z test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (1.091s) 2022-03-17T21:35:24.2733491Z test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-03-17T21:35:24.2788339Z test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-03-17T21:35:24.2844764Z test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.006s) 2022-03-17T21:35:24.2891851Z test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-03-17T21:35:24.2939115Z test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-03-17T21:35:24.2973198Z test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.003s) 2022-03-17T21:35:24.3012819Z test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-03-17T21:35:24.3052893Z test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s) 2022-03-17T21:35:24.3116741Z test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.006s) 2022-03-17T21:35:24.3168044Z test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.005s) 2022-03-17T21:35:24.3188191Z test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.002s) 2022-03-17T21:35:24.3188576Z 2022-03-17T21:35:24.3189122Z ---------------------------------------------------------------------- 2022-03-17T21:35:24.3189453Z Ran 12 tests in 1.142s 2022-03-17T21:35:24.3189626Z 2022-03-17T21:35:24.3189705Z OK 2022-03-17T21:35:24.3189731Z 2022-03-17T21:35:24.3189861Z Generating XML reports... 2022-03-17T21:35:24.3237207Z Generated XML report: test-reports/dist-gloo/distributed.test_nccl/TEST-TestNCCLCUDA-20220317213523.xml 2022-03-17T21:35:24.5949147Z Running distributed/test_pg_wrapper ... [2022-03-17 21:35:24.594390] 2022-03-17T21:35:24.5949783Z Executing ['/opt/conda/bin/python', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 21:35:24.594499] 2022-03-17T21:35:25.3847908Z 2022-03-17T21:35:25.3848271Z 2022-03-17T21:35:25.4790921Z Running distributed/test_store ... [2022-03-17 21:35:25.478649] 2022-03-17T21:35:25.4791868Z Executing ['/opt/conda/bin/python', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-03-17 21:35:25.478759] 2022-03-17T21:35:26.2651233Z test_compare_set (__main__.FileStoreTest) 2022-03-17T21:35:26.2651910Z test_set_get (__main__.FileStoreTest) 2022-03-17T21:35:26.2652460Z test_compare_set (__main__.PrefixFileStoreTest) 2022-03-17T21:35:26.2653070Z test_set_get (__main__.PrefixFileStoreTest) 2022-03-17T21:35:26.2653606Z test_compare_set (__main__.PrefixTCPStoreTest) 2022-03-17T21:35:26.2653938Z test_set_get (__main__.PrefixTCPStoreTest) 2022-03-17T21:35:26.2654246Z test_set_get (__main__.PythonStoreTest) 2022-03-17T21:35:26.2654793Z test_nominal (__main__.RendezvousEnvTest) 2022-03-17T21:35:26.2655151Z test_common_errors (__main__.RendezvousFileTest) 2022-03-17T21:35:26.2655487Z test_nominal (__main__.RendezvousFileTest) 2022-03-17T21:35:26.2656054Z test_unknown_handler (__main__.RendezvousTest) 2022-03-17T21:35:26.2656413Z test_address_already_in_use (__main__.TCPStoreTest) 2022-03-17T21:35:26.2656739Z test_compare_set (__main__.TCPStoreTest) 2022-03-17T21:35:26.2657080Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) 2022-03-17T21:35:26.2657429Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) 2022-03-17T21:35:26.2657804Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) 2022-03-17T21:35:26.2658146Z test_multitenancy (__main__.TCPStoreTest) 2022-03-17T21:35:26.2658443Z test_numkeys_delkeys (__main__.TCPStoreTest) 2022-03-17T21:35:26.2658744Z test_set_get (__main__.TCPStoreTest) 2022-03-17T21:35:27.0537997Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:27.0552163Z 2022-03-17T21:35:27.0552311Z Running tests... 2022-03-17T21:35:27.0553231Z ---------------------------------------------------------------------- 2022-03-17T21:35:28.6035776Z test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:28.6223004Z ok (1.567s) 2022-03-17T21:35:28.6224669Z 2022-03-17T21:35:28.6225270Z ---------------------------------------------------------------------- 2022-03-17T21:35:28.6225656Z Ran 1 test in 1.567s 2022-03-17T21:35:28.6225817Z 2022-03-17T21:35:28.6225897Z OK 2022-03-17T21:35:28.6226021Z 2022-03-17T21:35:28.6226125Z Generating XML reports... 2022-03-17T21:35:28.6258884Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-FileStoreTest-20220317213527.xml 2022-03-17T21:35:29.6331140Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:29.6345377Z 2022-03-17T21:35:29.6345659Z Running tests... 2022-03-17T21:35:29.6346143Z ---------------------------------------------------------------------- 2022-03-17T21:35:31.1717271Z test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:31.1889970Z ok (1.554s) 2022-03-17T21:35:31.1891055Z 2022-03-17T21:35:31.1891508Z ---------------------------------------------------------------------- 2022-03-17T21:35:31.1891842Z Ran 1 test in 1.555s 2022-03-17T21:35:31.1892001Z 2022-03-17T21:35:31.1892093Z OK 2022-03-17T21:35:31.1892218Z 2022-03-17T21:35:31.1892335Z Generating XML reports... 2022-03-17T21:35:31.1927639Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-FileStoreTest-20220317213529.xml 2022-03-17T21:35:32.1873346Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:32.1886728Z 2022-03-17T21:35:32.1887001Z Running tests... 2022-03-17T21:35:32.1887466Z ---------------------------------------------------------------------- 2022-03-17T21:35:33.7264162Z test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:33.7451433Z ok (1.556s) 2022-03-17T21:35:33.7453817Z 2022-03-17T21:35:33.7454639Z ---------------------------------------------------------------------- 2022-03-17T21:35:33.7454981Z Ran 1 test in 1.557s 2022-03-17T21:35:33.7455398Z 2022-03-17T21:35:33.7455470Z OK 2022-03-17T21:35:33.7455589Z 2022-03-17T21:35:33.7455707Z Generating XML reports... 2022-03-17T21:35:33.7489047Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PrefixFileStoreTest-20220317213532.xml 2022-03-17T21:35:34.7320856Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:34.7334282Z 2022-03-17T21:35:34.7334424Z Running tests... 2022-03-17T21:35:34.7335230Z ---------------------------------------------------------------------- 2022-03-17T21:35:36.2761540Z test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:36.2930372Z ok (1.559s) 2022-03-17T21:35:36.2932372Z 2022-03-17T21:35:36.2933165Z ---------------------------------------------------------------------- 2022-03-17T21:35:36.2933563Z Ran 1 test in 1.560s 2022-03-17T21:35:36.2933730Z 2022-03-17T21:35:36.2933822Z OK 2022-03-17T21:35:36.2933957Z 2022-03-17T21:35:36.2934047Z Generating XML reports... 2022-03-17T21:35:36.2966321Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PrefixFileStoreTest-20220317213534.xml 2022-03-17T21:35:37.2774731Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:37.2790155Z 2022-03-17T21:35:37.2790547Z Running tests... 2022-03-17T21:35:37.2791303Z ---------------------------------------------------------------------- 2022-03-17T21:35:38.8195394Z test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:38.8394544Z ok (1.560s) 2022-03-17T21:35:38.8397659Z 2022-03-17T21:35:38.8398546Z ---------------------------------------------------------------------- 2022-03-17T21:35:38.8398966Z Ran 1 test in 1.561s 2022-03-17T21:35:38.8399145Z 2022-03-17T21:35:38.8399236Z OK 2022-03-17T21:35:38.8399354Z 2022-03-17T21:35:38.8399485Z Generating XML reports... 2022-03-17T21:35:38.8432876Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PrefixTCPStoreTest-20220317213537.xml 2022-03-17T21:35:39.7884763Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:39.7900000Z 2022-03-17T21:35:39.7900538Z Running tests... 2022-03-17T21:35:39.7901305Z ---------------------------------------------------------------------- 2022-03-17T21:35:41.3336284Z test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:41.3525580Z ok (1.562s) 2022-03-17T21:35:41.3528293Z 2022-03-17T21:35:41.3528772Z ---------------------------------------------------------------------- 2022-03-17T21:35:41.3529169Z Ran 1 test in 1.563s 2022-03-17T21:35:41.3529265Z 2022-03-17T21:35:41.3529365Z OK 2022-03-17T21:35:41.3529515Z 2022-03-17T21:35:41.3529644Z Generating XML reports... 2022-03-17T21:35:41.3563585Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PrefixTCPStoreTest-20220317213539.xml 2022-03-17T21:35:42.3511392Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:42.3525078Z 2022-03-17T21:35:42.3525345Z Running tests... 2022-03-17T21:35:42.3525791Z ---------------------------------------------------------------------- 2022-03-17T21:35:43.8888997Z test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:43.9041658Z ok (1.551s) 2022-03-17T21:35:43.9042383Z 2022-03-17T21:35:43.9042927Z ---------------------------------------------------------------------- 2022-03-17T21:35:43.9043275Z Ran 1 test in 1.552s 2022-03-17T21:35:43.9043427Z 2022-03-17T21:35:43.9043519Z OK 2022-03-17T21:35:43.9043656Z 2022-03-17T21:35:43.9043756Z Generating XML reports... 2022-03-17T21:35:43.9077665Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-PythonStoreTest-20220317213542.xml 2022-03-17T21:35:44.9007394Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:44.9021505Z 2022-03-17T21:35:44.9021875Z Running tests... 2022-03-17T21:35:44.9022400Z ---------------------------------------------------------------------- 2022-03-17T21:35:46.4268614Z test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:46.4437861Z ok (1.541s) 2022-03-17T21:35:46.4438371Z 2022-03-17T21:35:46.4438870Z ---------------------------------------------------------------------- 2022-03-17T21:35:46.4439209Z Ran 1 test in 1.542s 2022-03-17T21:35:46.4439387Z 2022-03-17T21:35:46.4439480Z OK 2022-03-17T21:35:46.4439643Z 2022-03-17T21:35:46.4439715Z Generating XML reports... 2022-03-17T21:35:46.4473826Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-RendezvousEnvTest-20220317213544.xml 2022-03-17T21:35:47.4314501Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:47.4327874Z 2022-03-17T21:35:47.4328351Z Running tests... 2022-03-17T21:35:47.4328833Z ---------------------------------------------------------------------- 2022-03-17T21:35:48.9557734Z test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:48.9712710Z ok (1.538s) 2022-03-17T21:35:48.9713007Z 2022-03-17T21:35:48.9713457Z ---------------------------------------------------------------------- 2022-03-17T21:35:48.9713758Z Ran 1 test in 1.539s 2022-03-17T21:35:48.9713934Z 2022-03-17T21:35:48.9714018Z OK 2022-03-17T21:35:48.9714137Z 2022-03-17T21:35:48.9714264Z Generating XML reports... 2022-03-17T21:35:48.9748013Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-RendezvousFileTest-20220317213547.xml 2022-03-17T21:35:49.9595555Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:49.9609547Z 2022-03-17T21:35:49.9609701Z Running tests... 2022-03-17T21:35:49.9610410Z ---------------------------------------------------------------------- 2022-03-17T21:35:51.5038394Z test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:51.5206785Z ok (1.559s) 2022-03-17T21:35:51.5207170Z 2022-03-17T21:35:51.5207676Z ---------------------------------------------------------------------- 2022-03-17T21:35:51.5208022Z Ran 1 test in 1.560s 2022-03-17T21:35:51.5208179Z 2022-03-17T21:35:51.5208279Z OK 2022-03-17T21:35:51.5208417Z 2022-03-17T21:35:51.5208528Z Generating XML reports... 2022-03-17T21:35:51.5244676Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-RendezvousFileTest-20220317213549.xml 2022-03-17T21:35:52.5173148Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:52.5187054Z 2022-03-17T21:35:52.5187339Z Running tests... 2022-03-17T21:35:52.5187701Z ---------------------------------------------------------------------- 2022-03-17T21:35:54.0484634Z test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:54.0636090Z ok (1.545s) 2022-03-17T21:35:54.0636521Z 2022-03-17T21:35:54.0636969Z ---------------------------------------------------------------------- 2022-03-17T21:35:54.0637337Z Ran 1 test in 1.545s 2022-03-17T21:35:54.0637507Z 2022-03-17T21:35:54.0637586Z OK 2022-03-17T21:35:54.0637719Z 2022-03-17T21:35:54.0637854Z Generating XML reports... 2022-03-17T21:35:54.0670965Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-RendezvousTest-20220317213552.xml 2022-03-17T21:35:55.0575029Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:55.0589186Z 2022-03-17T21:35:55.0589571Z Running tests... 2022-03-17T21:35:55.0589952Z ---------------------------------------------------------------------- 2022-03-17T21:35:56.6052503Z test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:56.6215103Z [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:33299 (errno: 98 - Address already in use). 2022-03-17T21:35:56.6233264Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:33299 (errno: 98 - Address already in use). 2022-03-17T21:35:56.6234036Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address. 2022-03-17T21:35:56.6238312Z ok (1.565s) 2022-03-17T21:35:56.6239205Z 2022-03-17T21:35:56.6239589Z ---------------------------------------------------------------------- 2022-03-17T21:35:56.6239902Z Ran 1 test in 1.565s 2022-03-17T21:35:56.6240105Z 2022-03-17T21:35:56.6240177Z OK 2022-03-17T21:35:56.6240312Z 2022-03-17T21:35:56.6240427Z Generating XML reports... 2022-03-17T21:35:56.6274155Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213555.xml 2022-03-17T21:35:57.6172060Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:35:57.6187575Z 2022-03-17T21:35:57.6187776Z Running tests... 2022-03-17T21:35:57.6188231Z ---------------------------------------------------------------------- 2022-03-17T21:35:59.1463460Z test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:35:59.1664531Z ok (1.548s) 2022-03-17T21:35:59.1664868Z 2022-03-17T21:35:59.1665315Z ---------------------------------------------------------------------- 2022-03-17T21:35:59.1665634Z Ran 1 test in 1.548s 2022-03-17T21:35:59.1665820Z 2022-03-17T21:35:59.1665923Z OK 2022-03-17T21:35:59.1666036Z 2022-03-17T21:35:59.1666171Z Generating XML reports... 2022-03-17T21:35:59.1700837Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213557.xml 2022-03-17T21:36:00.1556011Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:36:00.1570221Z 2022-03-17T21:36:00.1570650Z Running tests... 2022-03-17T21:36:00.1571197Z ---------------------------------------------------------------------- 2022-03-17T21:36:01.6903659Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:36:01.7074196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 2022-03-17T21:36:01.7075241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. 2022-03-17T21:36:01.8121197Z ok (1.655s) 2022-03-17T21:36:01.8121394Z 2022-03-17T21:36:01.8121954Z ---------------------------------------------------------------------- 2022-03-17T21:36:01.8122531Z Ran 1 test in 1.655s 2022-03-17T21:36:01.8122701Z 2022-03-17T21:36:01.8122791Z OK 2022-03-17T21:36:01.8122937Z 2022-03-17T21:36:01.8123058Z Generating XML reports... 2022-03-17T21:36:01.8159683Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213600.xml 2022-03-17T21:36:02.8270877Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:36:02.8284888Z 2022-03-17T21:36:02.8285309Z Running tests... 2022-03-17T21:36:02.8285795Z ---------------------------------------------------------------------- 2022-03-17T21:36:04.3741137Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:36:04.3970277Z ok (1.568s) 2022-03-17T21:36:04.3970626Z 2022-03-17T21:36:04.3971049Z ---------------------------------------------------------------------- 2022-03-17T21:36:04.3971372Z Ran 1 test in 1.569s 2022-03-17T21:36:04.3971538Z 2022-03-17T21:36:04.3971637Z OK 2022-03-17T21:36:04.3971775Z 2022-03-17T21:36:04.3971886Z Generating XML reports... 2022-03-17T21:36:04.4006270Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213602.xml 2022-03-17T21:36:05.3898981Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:36:05.3913076Z 2022-03-17T21:36:05.3913512Z Running tests... 2022-03-17T21:36:05.3913989Z ---------------------------------------------------------------------- 2022-03-17T21:36:06.9224830Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:36:06.9488880Z ok (1.557s) 2022-03-17T21:36:06.9489074Z 2022-03-17T21:36:06.9489454Z ---------------------------------------------------------------------- 2022-03-17T21:36:06.9489808Z Ran 1 test in 1.558s 2022-03-17T21:36:06.9489968Z 2022-03-17T21:36:06.9490059Z OK 2022-03-17T21:36:06.9490172Z 2022-03-17T21:36:06.9490304Z Generating XML reports... 2022-03-17T21:36:06.9525105Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213605.xml 2022-03-17T21:36:07.9467624Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:36:07.9481436Z 2022-03-17T21:36:07.9481824Z Running tests... 2022-03-17T21:36:07.9482617Z ---------------------------------------------------------------------- 2022-03-17T21:36:09.4761860Z test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:36:09.4933779Z ok (1.545s) 2022-03-17T21:36:09.4933935Z 2022-03-17T21:36:09.4934306Z ---------------------------------------------------------------------- 2022-03-17T21:36:09.4934673Z Ran 1 test in 1.545s 2022-03-17T21:36:09.4934836Z 2022-03-17T21:36:09.4934909Z OK 2022-03-17T21:36:09.4935045Z 2022-03-17T21:36:09.4935169Z Generating XML reports... 2022-03-17T21:36:09.4969894Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213607.xml 2022-03-17T21:36:10.4433911Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:36:10.4447656Z 2022-03-17T21:36:10.4448080Z Running tests... 2022-03-17T21:36:10.4448572Z ---------------------------------------------------------------------- 2022-03-17T21:36:12.0032072Z test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:36:14.0478506Z ok (3.603s) 2022-03-17T21:36:14.0478717Z 2022-03-17T21:36:14.0479130Z ---------------------------------------------------------------------- 2022-03-17T21:36:14.0479502Z Ran 1 test in 3.603s 2022-03-17T21:36:14.0479688Z 2022-03-17T21:36:14.0479788Z OK 2022-03-17T21:36:14.0479909Z 2022-03-17T21:36:14.0480048Z Generating XML reports... 2022-03-17T21:36:14.0521225Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213610.xml 2022-03-17T21:36:15.0739526Z Test results will be stored in test-reports/dist-gloo/distributed.test_store 2022-03-17T21:36:15.0753604Z 2022-03-17T21:36:15.0754052Z Running tests... 2022-03-17T21:36:15.0754539Z ---------------------------------------------------------------------- 2022-03-17T21:36:16.6155601Z test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init 2022-03-17T21:36:16.6348325Z ok (1.559s) 2022-03-17T21:36:16.6348622Z 2022-03-17T21:36:16.6349129Z ---------------------------------------------------------------------- 2022-03-17T21:36:16.6349510Z Ran 1 test in 1.560s 2022-03-17T21:36:16.6349691Z 2022-03-17T21:36:16.6349777Z OK 2022-03-17T21:36:16.6349871Z 2022-03-17T21:36:16.6350064Z Generating XML reports... 2022-03-17T21:36:16.6384091Z Generated XML report: test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213615.xml 2022-03-17T21:36:17.7918115Z 2022-03-17T21:36:17.7918652Z real 142m2.121s 2022-03-17T21:36:17.7918932Z user 250m42.582s 2022-03-17T21:36:17.7919173Z sys 173m31.790s 2022-03-17T21:36:17.7919400Z + assert_git_not_dirty 2022-03-17T21:36:17.7919936Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *rocm* ]] 2022-03-17T21:36:17.7920426Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *xla* ]] 2022-03-17T21:36:17.7922726Z ++ git status --porcelain 2022-03-17T21:36:18.5738644Z + git_status= 2022-03-17T21:36:18.5739018Z + [[ -n '' ]] 2022-03-17T21:36:18.5739506Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed == *cuda* ]] 2022-03-17T21:36:18.5739919Z + echo 'Testing distributed C++ tests' 2022-03-17T21:36:18.5740271Z Testing distributed C++ tests 2022-03-17T21:36:18.5743255Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-03-17T21:36:18.5755579Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.7/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-03-17T21:36:18.5764892Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed 2022-03-17T21:36:18.5765366Z + mkdir -p test/test-reports/cpp-distributed/test_distributed 2022-03-17T21:36:18.5779400Z + /opt/conda/lib/python3.7/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml 2022-03-17T21:36:18.7041372Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2022-03-17T21:36:18.7041983Z [==========] Running 4 tests from 1 test suite. 2022-03-17T21:36:18.7042373Z [----------] Global test environment set-up. 2022-03-17T21:36:18.7042729Z [----------] 4 tests from FileStoreTest 2022-03-17T21:36:18.7043123Z [ RUN ] FileStoreTest.testGetAndSet 2022-03-17T21:36:18.7047105Z [ OK ] FileStoreTest.testGetAndSet (0 ms) 2022-03-17T21:36:18.7047606Z [ RUN ] FileStoreTest.testGetAndSetWithPrefix 2022-03-17T21:36:18.7051595Z [ OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms) 2022-03-17T21:36:18.7052073Z [ RUN ] FileStoreTest.testStressStore 2022-03-17T21:36:18.7247496Z [ OK ] FileStoreTest.testStressStore (19 ms) 2022-03-17T21:36:18.7247983Z [ RUN ] FileStoreTest.testStressStoreWithPrefix 2022-03-17T21:36:18.7447523Z [ OK ] FileStoreTest.testStressStoreWithPrefix (19 ms) 2022-03-17T21:36:18.7448045Z [----------] 4 tests from FileStoreTest (40 ms total) 2022-03-17T21:36:18.7448326Z 2022-03-17T21:36:18.7448645Z [----------] Global test environment tear-down 2022-03-17T21:36:18.7449508Z [==========] 4 tests from 1 test suite ran. (40 ms total) 2022-03-17T21:36:18.7449931Z [ PASSED ] 4 tests. 2022-03-17T21:36:18.7680587Z + /opt/conda/lib/python3.7/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml 2022-03-17T21:36:18.8930700Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2022-03-17T21:36:18.8931359Z [==========] Running 4 tests from 1 test suite. 2022-03-17T21:36:18.8931726Z [----------] Global test environment set-up. 2022-03-17T21:36:18.8932141Z [----------] 4 tests from HashStoreTest 2022-03-17T21:36:18.8932513Z [ RUN ] HashStoreTest.testGetAndSet 2022-03-17T21:36:18.9934298Z [ OK ] HashStoreTest.testGetAndSet (100 ms) 2022-03-17T21:36:18.9934780Z [ RUN ] HashStoreTest.testGetAndSetWithPrefix 2022-03-17T21:36:19.0938502Z [ OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms) 2022-03-17T21:36:19.0938979Z [ RUN ] HashStoreTest.testStressStore 2022-03-17T21:36:19.0946280Z [ OK ] HashStoreTest.testStressStore (0 ms) 2022-03-17T21:36:19.0946784Z [ RUN ] HashStoreTest.testStressStoreWithPrefix 2022-03-17T21:36:19.0953281Z [ OK ] HashStoreTest.testStressStoreWithPrefix (0 ms) 2022-03-17T21:36:19.0953807Z [----------] 4 tests from HashStoreTest (202 ms total) 2022-03-17T21:36:19.0954033Z 2022-03-17T21:36:19.0954254Z [----------] Global test environment tear-down 2022-03-17T21:36:19.0956819Z [==========] 4 tests from 1 test suite ran. (202 ms total) 2022-03-17T21:36:19.0957215Z [ PASSED ] 4 tests. 2022-03-17T21:36:19.1197920Z + /opt/conda/lib/python3.7/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml 2022-03-17T21:36:19.2464581Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2022-03-17T21:36:19.2465184Z [==========] Running 11 tests from 1 test suite. 2022-03-17T21:36:19.2465568Z [----------] Global test environment set-up. 2022-03-17T21:36:19.2466138Z [----------] 11 tests from TCPStoreTest 2022-03-17T21:36:19.2466571Z [ RUN ] TCPStoreTest.testHelper 2022-03-17T21:36:20.2380847Z [ OK ] TCPStoreTest.testHelper (991 ms) 2022-03-17T21:36:20.2381375Z [ RUN ] TCPStoreTest.testHelperPrefix 2022-03-17T21:36:21.2020706Z [ OK ] TCPStoreTest.testHelperPrefix (963 ms) 2022-03-17T21:36:21.2021234Z [ RUN ] TCPStoreTest.testWatchKeyCallback 2022-03-17T21:36:21.2160582Z [ OK ] TCPStoreTest.testWatchKeyCallback (14 ms) 2022-03-17T21:36:21.2161087Z [ RUN ] TCPStoreTest.testWatchKeyCallbackWithPrefix 2022-03-17T21:36:21.2297692Z [ OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (13 ms) 2022-03-17T21:36:21.2298211Z [ RUN ] TCPStoreTest.testKeyEmptyUpdate 2022-03-17T21:36:21.4347314Z [ OK ] TCPStoreTest.testKeyEmptyUpdate (204 ms) 2022-03-17T21:36:21.4347668Z [ RUN ] TCPStoreTest.testKeyUpdate 2022-03-17T21:36:21.4353334Z [ OK ] TCPStoreTest.testKeyUpdate (0 ms) 2022-03-17T21:36:21.4353735Z [ RUN ] TCPStoreTest.testKeyCreate 2022-03-17T21:36:21.4357976Z [ OK ] TCPStoreTest.testKeyCreate (0 ms) 2022-03-17T21:36:21.4358359Z [ RUN ] TCPStoreTest.testKeyAdd 2022-03-17T21:36:21.4362608Z [ OK ] TCPStoreTest.testKeyAdd (0 ms) 2022-03-17T21:36:21.4363010Z [ RUN ] TCPStoreTest.testKeyDelete 2022-03-17T21:36:21.6425923Z [ OK ] TCPStoreTest.testKeyDelete (206 ms) 2022-03-17T21:36:21.6426357Z [ RUN ] TCPStoreTest.testCleanShutdown 2022-03-17T21:36:21.6433177Z [ OK ] TCPStoreTest.testCleanShutdown (0 ms) 2022-03-17T21:36:21.6433620Z [ RUN ] TCPStoreTest.testMultiTenantStores 2022-03-17T21:36:21.6445669Z [ OK ] TCPStoreTest.testMultiTenantStores (1 ms) 2022-03-17T21:36:21.6446442Z [----------] 11 tests from TCPStoreTest (2398 ms total) 2022-03-17T21:36:21.6446655Z 2022-03-17T21:36:21.6446822Z [----------] Global test environment tear-down 2022-03-17T21:36:21.6449529Z [==========] 11 tests from 1 test suite ran. (2398 ms total) 2022-03-17T21:36:21.6449907Z [ PASSED ] 11 tests. 2022-03-17T21:36:21.6725129Z ++ command -v mpiexec 2022-03-17T21:36:21.6728221Z + MPIEXEC=/usr/bin/mpiexec 2022-03-17T21:36:21.6728779Z + [[ -n /usr/bin/mpiexec ]] 2022-03-17T21:36:21.6729107Z + [[ -z true ]] 2022-03-17T21:36:21.6729681Z + /opt/conda/lib/python3.7/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml 2022-03-17T21:36:21.8327989Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2022-03-17T21:36:21.8329252Z [==========] Running 12 tests from 1 test suite. 2022-03-17T21:36:21.8329727Z [----------] Global test environment set-up. 2022-03-17T21:36:21.8330151Z [----------] 12 tests from ProcessGroupGlooTest 2022-03-17T21:36:21.8330627Z [ RUN ] ProcessGroupGlooTest.testSIGSTOPException 2022-03-17T21:36:22.8873774Z [ OK ] ProcessGroupGlooTest.testSIGSTOPException (1054 ms) 2022-03-17T21:36:22.8874772Z [ RUN ] ProcessGroupGlooTest.testSIGKILLException 2022-03-17T21:36:22.9208059Z [ OK ] ProcessGroupGlooTest.testSIGKILLException (33 ms) 2022-03-17T21:36:22.9209079Z [ RUN ] ProcessGroupGlooTest.testAllReduceCPU 2022-03-17T21:36:23.1987450Z [ OK ] ProcessGroupGlooTest.testAllReduceCPU (277 ms) 2022-03-17T21:36:23.1988399Z [ RUN ] ProcessGroupGlooTest.testBroadcastCPU 2022-03-17T21:36:23.2412222Z [ OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms) 2022-03-17T21:36:23.2413235Z [ RUN ] ProcessGroupGlooTest.testAllToAllCPU 2022-03-17T21:36:23.3743667Z [ OK ] ProcessGroupGlooTest.testAllToAllCPU (133 ms) 2022-03-17T21:36:23.3744765Z [ RUN ] ProcessGroupGlooTest.testBarrier 2022-03-17T21:36:23.4261005Z [ OK ] ProcessGroupGlooTest.testBarrier (51 ms) 2022-03-17T21:36:23.4262015Z [ RUN ] ProcessGroupGlooTest.testMonitoredBarrier 2022-03-17T21:36:24.4569019Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms 2022-03-17T21:36:24.4770791Z [ OK ] ProcessGroupGlooTest.testMonitoredBarrier (1050 ms) 2022-03-17T21:36:24.4771802Z [ RUN ] ProcessGroupGlooTest.testSequenceNumInit 2022-03-17T21:36:24.6197130Z [ OK ] ProcessGroupGlooTest.testSequenceNumInit (142 ms) 2022-03-17T21:36:24.6198140Z [ RUN ] ProcessGroupGlooTest.testSend 2022-03-17T21:36:24.6713917Z [ OK ] ProcessGroupGlooTest.testSend (51 ms) 2022-03-17T21:36:24.6714823Z [ RUN ] ProcessGroupGlooTest.testRecv 2022-03-17T21:36:24.7231068Z [ OK ] ProcessGroupGlooTest.testRecv (51 ms) 2022-03-17T21:36:24.7232027Z [ RUN ] ProcessGroupGlooTest.testStoreSetGet 2022-03-17T21:36:24.7640419Z [ OK ] ProcessGroupGlooTest.testStoreSetGet (40 ms) 2022-03-17T21:36:24.7641396Z [ RUN ] ProcessGroupGlooTest.testWaitDelay 2022-03-17T21:36:24.9157217Z [ OK ] ProcessGroupGlooTest.testWaitDelay (151 ms) 2022-03-17T21:36:24.9158294Z [----------] 12 tests from ProcessGroupGlooTest (3082 ms total) 2022-03-17T21:36:24.9158565Z 2022-03-17T21:36:24.9158804Z [----------] Global test environment tear-down 2022-03-17T21:36:24.9161131Z [==========] 12 tests from 1 test suite ran. (3082 ms total) 2022-03-17T21:36:24.9161920Z [ PASSED ] 12 tests. 2022-03-17T21:36:24.9574755Z + /opt/conda/lib/python3.7/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml 2022-03-17T21:36:25.1192860Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2022-03-17T21:36:25.1194076Z [==========] Running 11 tests from 1 test suite. 2022-03-17T21:36:25.1194574Z [----------] Global test environment set-up. 2022-03-17T21:36:25.1194978Z [----------] 11 tests from ProcessGroupNCCLTest 2022-03-17T21:36:25.1195427Z [ RUN ] ProcessGroupNCCLTest.testAllreduce 2022-03-17T21:36:32.7710530Z [ OK ] ProcessGroupNCCLTest.testAllreduce (7651 ms) 2022-03-17T21:36:32.7711084Z [ RUN ] ProcessGroupNCCLTest.testBroadcast 2022-03-17T21:36:32.8592987Z [ OK ] ProcessGroupNCCLTest.testBroadcast (88 ms) 2022-03-17T21:36:32.8593488Z [ RUN ] ProcessGroupNCCLTest.testReduce 2022-03-17T21:36:32.9507623Z [ OK ] ProcessGroupNCCLTest.testReduce (91 ms) 2022-03-17T21:36:32.9508155Z [ RUN ] ProcessGroupNCCLTest.testAllgather 2022-03-17T21:36:33.0396908Z [ OK ] ProcessGroupNCCLTest.testAllgather (88 ms) 2022-03-17T21:36:33.0397472Z [ RUN ] ProcessGroupNCCLTest.testAllgatherBase 2022-03-17T21:36:33.1207220Z [ OK ] ProcessGroupNCCLTest.testAllgatherBase (81 ms) 2022-03-17T21:36:33.1207746Z [ RUN ] ProcessGroupNCCLTest.testReduceScatter 2022-03-17T21:36:33.2085301Z [ OK ] ProcessGroupNCCLTest.testReduceScatter (87 ms) 2022-03-17T21:36:33.2086087Z [ RUN ] ProcessGroupNCCLTest.testSequenceNumInit 2022-03-17T21:36:33.2656602Z [ OK ] ProcessGroupNCCLTest.testSequenceNumInit (57 ms) 2022-03-17T21:36:33.2657832Z [ RUN ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout 2022-03-17T21:36:36.2674336Z [ OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3001 ms) 2022-03-17T21:36:36.2675105Z [ RUN ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException 2022-03-17T21:36:39.2684474Z [ OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms) 2022-03-17T21:36:39.2685265Z [ RUN ] ProcessGroupNCCLTest.testReduceScatterBase 2022-03-17T21:36:39.3488396Z [ OK ] ProcessGroupNCCLTest.testReduceScatterBase (80 ms) 2022-03-17T21:36:39.3488918Z [ RUN ] ProcessGroupNCCLTest.testBackendName 2022-03-17T21:36:39.3924422Z [ OK ] ProcessGroupNCCLTest.testBackendName (43 ms) 2022-03-17T21:36:39.3924977Z [----------] 11 tests from ProcessGroupNCCLTest (14273 ms total) 2022-03-17T21:36:39.3925191Z 2022-03-17T21:36:39.3925423Z [----------] Global test environment tear-down 2022-03-17T21:36:39.3928599Z [==========] 11 tests from 1 test suite ran. (14273 ms total) 2022-03-17T21:36:39.3928972Z [ PASSED ] 11 tests. 2022-03-17T21:36:40.1164744Z + /opt/conda/lib/python3.7/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml 2022-03-17T21:36:40.2826303Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc 2022-03-17T21:36:40.2826924Z [==========] Running 3 tests from 1 test suite. 2022-03-17T21:36:40.2827300Z [----------] Global test environment set-up. 2022-03-17T21:36:40.2827756Z [----------] 3 tests from ProcessGroupNCCLErrorsTest 2022-03-17T21:36:40.2828283Z [ RUN ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking 2022-03-17T21:36:46.2908091Z [ OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (6007 ms) 2022-03-17T21:36:46.2908782Z [ RUN ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking 2022-03-17T21:36:49.3383671Z [ OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3047 ms) 2022-03-17T21:36:49.3384313Z [ RUN ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking 2022-03-17T21:36:49.3898506Z [ OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (51 ms) 2022-03-17T21:36:49.3899381Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (9107 ms total) 2022-03-17T21:36:49.3899646Z 2022-03-17T21:36:49.3899863Z [----------] Global test environment tear-down 2022-03-17T21:36:49.3900625Z [==========] 3 tests from 1 test suite ran. (9107 ms total) 2022-03-17T21:36:49.3901031Z [ PASSED ] 3 tests. 2022-03-17T21:36:50.1082010Z + test_rpc 2022-03-17T21:36:50.1082626Z + [[ linux-xenial-cuda11.3-py3.7-gcc7-distributed != *rocm* ]] 2022-03-17T21:36:50.1083039Z + echo 'Testing RPC C++ tests' 2022-03-17T21:36:50.1083312Z Testing RPC C++ tests 2022-03-17T21:36:50.1084866Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-03-17T21:36:50.1098802Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.7/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.7/site-packages/torch/bin 2022-03-17T21:36:50.1110923Z + ln -sf '/opt/conda/lib/python3.7/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.7/site-packages/torch/bin 2022-03-17T21:36:50.1120419Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc 2022-03-17T21:36:50.1120869Z + mkdir -p test/test-reports/cpp-rpc/test_rpc 2022-03-17T21:36:50.1134149Z + /opt/conda/lib/python3.7/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml 2022-03-17T21:36:54.0265604Z [==========] Running 8 tests from 3 test suites. 2022-03-17T21:36:54.0266078Z [----------] Global test environment set-up. 2022-03-17T21:36:54.0266457Z [----------] 4 tests from WireSerialize 2022-03-17T21:36:54.0266833Z [ RUN ] WireSerialize.Base 2022-03-17T21:36:54.0479548Z [ OK ] WireSerialize.Base (21 ms) 2022-03-17T21:36:54.0479953Z [ RUN ] WireSerialize.RecopySparseTensors 2022-03-17T21:36:54.0581871Z [ OK ] WireSerialize.RecopySparseTensors (10 ms) 2022-03-17T21:36:54.0582332Z [ RUN ] WireSerialize.CloneSparseTensors 2022-03-17T21:36:54.0678459Z [ OK ] WireSerialize.CloneSparseTensors (9 ms) 2022-03-17T21:36:54.0678849Z [ RUN ] WireSerialize.Errors 2022-03-17T21:36:54.0704696Z [ OK ] WireSerialize.Errors (2 ms) 2022-03-17T21:36:54.0705202Z [----------] 4 tests from WireSerialize (44 ms total) 2022-03-17T21:36:54.0705431Z 2022-03-17T21:36:54.0705637Z [----------] 1 test from TestE2ETensorPipe 2022-03-17T21:36:54.0706000Z [ RUN ] TestE2ETensorPipe.TestTrainingLoop 2022-03-17T21:36:54.8226862Z [W tensorpipe_agent.cpp:682] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356) 2022-03-17T21:36:54.8245520Z [ OK ] TestE2ETensorPipe.TestTrainingLoop (753 ms) 2022-03-17T21:36:54.8246027Z [----------] 1 test from TestE2ETensorPipe (754 ms total) 2022-03-17T21:36:54.8246238Z 2022-03-17T21:36:54.8246471Z [----------] 3 tests from TensorpipeSerialize 2022-03-17T21:36:54.8246874Z [ RUN ] TensorpipeSerialize.Base 2022-03-17T21:36:54.8247273Z [ OK ] TensorpipeSerialize.Base (0 ms) 2022-03-17T21:36:54.8247732Z [ RUN ] TensorpipeSerialize.RecopySparseTensors 2022-03-17T21:36:54.8343390Z [ OK ] TensorpipeSerialize.RecopySparseTensors (9 ms) 2022-03-17T21:36:54.8344347Z [ RUN ] TensorpipeSerialize.NoDeleterTensors 2022-03-17T21:36:54.8345278Z [ OK ] TensorpipeSerialize.NoDeleterTensors (0 ms) 2022-03-17T21:36:54.8346185Z [----------] 3 tests from TensorpipeSerialize (9 ms total) 2022-03-17T21:36:54.8346621Z 2022-03-17T21:36:54.8347054Z [----------] Global test environment tear-down 2022-03-17T21:36:54.8348115Z [==========] 8 tests from 3 test suites ran. (808 ms total) 2022-03-17T21:36:54.8348748Z [ PASSED ] 8 tests. 2022-03-17T21:36:54.8349059Z 2022-03-17T21:36:54.8349368Z  YOU HAVE 1 DISABLED TEST 2022-03-17T21:36:54.8349690Z 2022-03-17T21:36:55.3638551Z + cleanup 2022-03-17T21:36:55.3638798Z + retcode=0 2022-03-17T21:36:55.3639024Z + set +x 2022-03-17T21:36:55.3639236Z EXITED_USER_LAND 2022-03-17T21:36:55.3713762Z ##[group]Run # Ensure the working directory gets chowned back to the current user 2022-03-17T21:36:55.3714245Z # Ensure the working directory gets chowned back to the current user 2022-03-17T21:36:55.3714681Z docker run --rm -v "$(pwd)":/v -w /v "${ALPINE_IMAGE}" chown -R "$(id -u):$(id -g)" . 2022-03-17T21:36:55.3728181Z shell: /usr/bin/bash -e {0} 2022-03-17T21:36:55.3728566Z env: 2022-03-17T21:36:55.3728861Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:36:55.3729373Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:36:55.3729854Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:36:55.3730282Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:36:55.3730650Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:36:55.3730880Z IN_CI: 1 2022-03-17T21:36:55.3731099Z IS_GHA: 1 2022-03-17T21:36:55.3731359Z IN_WHEEL_TEST: 1 2022-03-17T21:36:55.3731650Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:36:55.3732040Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:36:55.3732353Z PR_LABELS: [] 2022-03-17T21:36:55.3732768Z GITHUB_TOKEN: *** 2022-03-17T21:36:55.3733039Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:36:55.3733279Z PR_NUMBER: 2022-03-17T21:36:55.3733560Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:36:55.3733831Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:36:55.3734102Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:36:55.3734605Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:36:55.3735136Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:36:55.3735455Z TEST_CONFIG: distributed 2022-03-17T21:36:55.3735692Z SHARD_NUMBER: 1 2022-03-17T21:36:55.3735930Z NUM_TEST_SHARDS: 1 2022-03-17T21:36:55.3736172Z PR_BODY: 2022-03-17T21:36:55.3736395Z GPU_FLAG: --gpus all 2022-03-17T21:36:55.3736637Z SHM_SIZE: 2g 2022-03-17T21:36:55.3736854Z ##[endgroup] 2022-03-17T21:36:56.9544432Z ##[group]Run python3 -m pip install junitparser==2.1.1 rich==10.9.0 2022-03-17T21:36:56.9544864Z python3 -m pip install junitparser==2.1.1 rich==10.9.0 2022-03-17T21:36:56.9557891Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T21:36:56.9558201Z env: 2022-03-17T21:36:56.9558489Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:36:56.9558994Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:36:56.9559470Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:36:56.9559891Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:36:56.9560257Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:36:56.9560483Z IN_CI: 1 2022-03-17T21:36:56.9560701Z IS_GHA: 1 2022-03-17T21:36:56.9560912Z IN_WHEEL_TEST: 1 2022-03-17T21:36:56.9561215Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:36:56.9561621Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:36:56.9561949Z PR_LABELS: [] 2022-03-17T21:36:56.9562576Z GITHUB_TOKEN: *** 2022-03-17T21:36:56.9562833Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:36:56.9563099Z PR_NUMBER: 2022-03-17T21:36:56.9563352Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:36:56.9563641Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:36:56.9563915Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:36:56.9564401Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:36:56.9564955Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:36:56.9565258Z TEST_CONFIG: distributed 2022-03-17T21:36:56.9565507Z SHARD_NUMBER: 1 2022-03-17T21:36:56.9565746Z NUM_TEST_SHARDS: 1 2022-03-17T21:36:56.9565964Z PR_BODY: 2022-03-17T21:36:56.9566201Z GPU_FLAG: --gpus all 2022-03-17T21:36:56.9566430Z SHM_SIZE: 2g 2022-03-17T21:36:56.9566664Z ##[endgroup] 2022-03-17T21:36:57.7288224Z Defaulting to user installation because normal site-packages is not writeable 2022-03-17T21:36:57.8259454Z Collecting junitparser==2.1.1 2022-03-17T21:36:57.8449653Z Downloading junitparser-2.1.1-py2.py3-none-any.whl (10 kB) 2022-03-17T21:36:58.0257828Z Collecting rich==10.9.0 2022-03-17T21:36:58.0404776Z Downloading rich-10.9.0-py3-none-any.whl (211 kB) 2022-03-17T21:36:58.0961826Z Collecting future 2022-03-17T21:36:58.1005842Z Downloading future-0.18.2.tar.gz (829 kB) 2022-03-17T21:36:58.6794161Z Collecting pygments<3.0.0,>=2.6.0 2022-03-17T21:36:58.6842440Z Downloading Pygments-2.11.2-py3-none-any.whl (1.1 MB) 2022-03-17T21:36:58.7655523Z Collecting colorama<0.5.0,>=0.4.0 2022-03-17T21:36:58.7703059Z Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB) 2022-03-17T21:36:58.8081861Z Collecting typing-extensions<4.0.0,>=3.7.4; python_version < "3.8" 2022-03-17T21:36:58.8125794Z Downloading typing_extensions-3.10.0.2-py3-none-any.whl (26 kB) 2022-03-17T21:36:58.8487205Z Collecting commonmark<0.10.0,>=0.9.0 2022-03-17T21:36:58.8528801Z Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB) 2022-03-17T21:36:58.8654329Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed. 2022-03-17T21:36:58.9181777Z Installing collected packages: future, junitparser, pygments, colorama, typing-extensions, commonmark, rich 2022-03-17T21:36:58.9191350Z Running setup.py install for future: started 2022-03-17T21:36:59.5680063Z Running setup.py install for future: finished with status 'done' 2022-03-17T21:36:59.5924464Z WARNING: The script junitparser is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-03-17T21:36:59.5925088Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-03-17T21:37:00.2023842Z WARNING: The script pygmentize is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-03-17T21:37:00.2024691Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-03-17T21:37:00.2864501Z WARNING: The script cmark is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-03-17T21:37:00.2865081Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-03-17T21:37:00.4633018Z Successfully installed colorama-0.4.4 commonmark-0.9.1 future-0.18.2 junitparser-2.1.1 pygments-2.11.2 rich-10.9.0 typing-extensions-3.10.0.2 2022-03-17T21:37:00.6669228Z ##[group]Run python3 tools/render_junit.py test/ 2022-03-17T21:37:00.6669616Z python3 tools/render_junit.py test/ 2022-03-17T21:37:00.6682829Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T21:37:00.6683130Z env: 2022-03-17T21:37:00.6683425Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:00.6683936Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:00.6684436Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:00.6684842Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:00.6685222Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:00.6685452Z IN_CI: 1 2022-03-17T21:37:00.6685675Z IS_GHA: 1 2022-03-17T21:37:00.6685910Z IN_WHEEL_TEST: 1 2022-03-17T21:37:00.6686199Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:00.6686627Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:00.6686939Z PR_LABELS: [] 2022-03-17T21:37:00.6687384Z GITHUB_TOKEN: *** 2022-03-17T21:37:00.6687657Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:00.6687898Z PR_NUMBER: 2022-03-17T21:37:00.6688173Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:00.6688446Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:00.6688723Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:00.6689209Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:00.6689763Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:00.6690227Z TEST_CONFIG: distributed 2022-03-17T21:37:00.6690463Z SHARD_NUMBER: 1 2022-03-17T21:37:00.6690705Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:00.6690927Z PR_BODY: 2022-03-17T21:37:00.6691170Z GPU_FLAG: --gpus all 2022-03-17T21:37:00.6691415Z SHM_SIZE: 2g 2022-03-17T21:37:00.6691658Z PYTHONIOENCODING: utf-8 2022-03-17T21:37:00.6691924Z ##[endgroup] 2022-03-17T21:37:00.9210715Z ✅ 1499 Passed 2022-03-17T21:37:00.9214870Z 💨 567 Skipped 2022-03-17T21:37:00.9219545Z 🚨 0 Failed 2022-03-17T21:37:00.9429629Z ##[group]Run # Remove any previous test jsons if they exist 2022-03-17T21:37:00.9430066Z # Remove any previous test jsons if they exist 2022-03-17T21:37:00.9430388Z rm -f test-jsons-*.zip 2022-03-17T21:37:00.9430704Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2022-03-17T21:37:00.9443859Z shell: /usr/bin/bash -e {0} 2022-03-17T21:37:00.9444118Z env: 2022-03-17T21:37:00.9444432Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:00.9444941Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:00.9445440Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:00.9445848Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:00.9446215Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:00.9446464Z IN_CI: 1 2022-03-17T21:37:00.9446666Z IS_GHA: 1 2022-03-17T21:37:00.9446943Z IN_WHEEL_TEST: 1 2022-03-17T21:37:00.9447236Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:00.9447629Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:00.9447937Z PR_LABELS: [] 2022-03-17T21:37:00.9448399Z GITHUB_TOKEN: *** 2022-03-17T21:37:00.9448680Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:00.9448921Z PR_NUMBER: 2022-03-17T21:37:00.9449197Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:00.9449475Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:00.9449756Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:00.9450264Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:00.9450804Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:00.9451131Z TEST_CONFIG: distributed 2022-03-17T21:37:00.9451367Z SHARD_NUMBER: 1 2022-03-17T21:37:00.9451613Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:00.9451840Z PR_BODY: 2022-03-17T21:37:00.9452087Z GPU_FLAG: --gpus all 2022-03-17T21:37:00.9452331Z SHM_SIZE: 2g 2022-03-17T21:37:00.9452665Z FILE_SUFFIX: test_distributed_1_1-distributed-1-1-linux.8xlarge.nvidia.gpu 2022-03-17T21:37:00.9453020Z ##[endgroup] 2022-03-17T21:37:00.9644755Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2022-03-17T21:37:00.9645470Z adding: test/.pytorch-slow-tests.json (deflated 63%) 2022-03-17T21:37:00.9648663Z adding: test/.pytorch-disabled-tests.json (deflated 82%) 2022-03-17T21:37:00.9682655Z ##[group]Run seemethere/upload-artifact-s3@v3 2022-03-17T21:37:00.9682983Z with: 2022-03-17T21:37:00.9683206Z retention-days: 14 2022-03-17T21:37:00.9683476Z if-no-files-found: warn 2022-03-17T21:37:00.9683732Z path: test-jsons-*.zip 2022-03-17T21:37:00.9683985Z name: artifact 2022-03-17T21:37:00.9684238Z s3-bucket: gha-artifacts 2022-03-17T21:37:00.9684481Z region: us-east-1 2022-03-17T21:37:00.9684709Z env: 2022-03-17T21:37:00.9684990Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:00.9685492Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:00.9685982Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:00.9686386Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:00.9686748Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:00.9686974Z IN_CI: 1 2022-03-17T21:37:00.9687333Z IS_GHA: 1 2022-03-17T21:37:00.9687588Z IN_WHEEL_TEST: 1 2022-03-17T21:37:00.9687879Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:00.9688276Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:00.9688607Z PR_LABELS: [] 2022-03-17T21:37:00.9688980Z GITHUB_TOKEN: *** 2022-03-17T21:37:00.9689250Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:00.9689494Z PR_NUMBER: 2022-03-17T21:37:00.9689777Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:00.9690074Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:00.9690330Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:00.9690835Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:00.9691370Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:00.9691694Z TEST_CONFIG: distributed 2022-03-17T21:37:00.9691947Z SHARD_NUMBER: 1 2022-03-17T21:37:00.9692179Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:00.9692418Z PR_BODY: 2022-03-17T21:37:00.9692641Z GPU_FLAG: --gpus all 2022-03-17T21:37:00.9692882Z SHM_SIZE: 2g 2022-03-17T21:37:00.9693096Z ##[endgroup] 2022-03-17T21:37:01.3668055Z With the provided path, there will be 1 file uploaded 2022-03-17T21:37:01.3668544Z Uploading to s3 prefix: pytorch/pytorch/2000264336/artifact 2022-03-17T21:37:01.3715341Z Starting upload of test-jsons-test_distributed_1_1-distributed-1-1-linux.8xlarge.nvidia.gpu.zip 2022-03-17T21:37:01.5218716Z Finished upload of test-jsons-test_distributed_1_1-distributed-1-1-linux.8xlarge.nvidia.gpu.zip 2022-03-17T21:37:01.5337456Z ##[group]Run # Remove any previous test reports if they exist 2022-03-17T21:37:01.5337903Z # Remove any previous test reports if they exist 2022-03-17T21:37:01.5338215Z rm -f test-reports-*.zip 2022-03-17T21:37:01.5338560Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' 2022-03-17T21:37:01.5352433Z shell: /usr/bin/bash -e {0} 2022-03-17T21:37:01.5352688Z env: 2022-03-17T21:37:01.5352984Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:01.5353494Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:01.5353975Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:01.5354397Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:01.5354763Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:01.5354993Z IN_CI: 1 2022-03-17T21:37:01.5355219Z IS_GHA: 1 2022-03-17T21:37:01.5355484Z IN_WHEEL_TEST: 1 2022-03-17T21:37:01.5355780Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:01.5356166Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:01.5356473Z PR_LABELS: [] 2022-03-17T21:37:01.5356891Z GITHUB_TOKEN: *** 2022-03-17T21:37:01.5357163Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:01.5357413Z PR_NUMBER: 2022-03-17T21:37:01.5357687Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:01.5357998Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:01.5358233Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:01.5358742Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:01.5359277Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:01.5359598Z TEST_CONFIG: distributed 2022-03-17T21:37:01.5359833Z SHARD_NUMBER: 1 2022-03-17T21:37:01.5360074Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:01.5360315Z PR_BODY: 2022-03-17T21:37:01.5360540Z GPU_FLAG: --gpus all 2022-03-17T21:37:01.5360782Z SHM_SIZE: 2g 2022-03-17T21:37:01.5361117Z FILE_SUFFIX: test_distributed_1_1-distributed-1-1-linux.8xlarge.nvidia.gpu 2022-03-17T21:37:01.5361467Z ##[endgroup] 2022-03-17T21:37:01.5576427Z adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220317191422.xml (deflated 59%) 2022-03-17T21:37:01.5577531Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220317191427.xml (deflated 74%) 2022-03-17T21:37:01.5578523Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220317191434.xml (deflated 60%) 2022-03-17T21:37:01.5579350Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220317191439.xml (deflated 60%) 2022-03-17T21:37:01.5580489Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220317191443.xml (deflated 60%) 2022-03-17T21:37:01.5581436Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220317191448.xml (deflated 68%) 2022-03-17T21:37:01.5582330Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220317191454.xml (deflated 68%) 2022-03-17T21:37:01.5583255Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220317191500.xml (deflated 44%) 2022-03-17T21:37:01.5584140Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_partial_tensor/TEST-TestPartialTensorReshard-20220317191503.xml (deflated 61%) 2022-03-17T21:37:01.5585131Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20220317191508.xml (deflated 42%) 2022-03-17T21:37:01.5586027Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20220317191508.xml (deflated 44%) 2022-03-17T21:37:01.5586852Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20220317191508.xml (deflated 59%) 2022-03-17T21:37:01.5587656Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20220317191508.xml (deflated 58%) 2022-03-17T21:37:01.5588512Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20220317191508.xml (deflated 60%) 2022-03-17T21:37:01.5589248Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20220317191508.xml (deflated 60%) 2022-03-17T21:37:01.5590102Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20220317191508.xml (deflated 89%) 2022-03-17T21:37:01.5590963Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20220317191508.xml (deflated 69%) 2022-03-17T21:37:01.5591841Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20220317191508.xml (deflated 84%) 2022-03-17T21:37:01.5592739Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20220317191508.xml (deflated 85%) 2022-03-17T21:37:01.5593680Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20220317191508.xml (deflated 61%) 2022-03-17T21:37:01.5594584Z adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220317191610.xml (deflated 61%) 2022-03-17T21:37:01.5595353Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220317191615.xml (deflated 65%) 2022-03-17T21:37:01.5596336Z adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220317191615.xml (deflated 78%) 2022-03-17T21:37:01.5597027Z adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220317191621.xml (deflated 79%) 2022-03-17T21:37:01.5597765Z adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220317191656.xml (deflated 62%) 2022-03-17T21:37:01.5598587Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220317191659.xml (deflated 55%) 2022-03-17T21:37:01.5599516Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220317191659.xml (deflated 80%) 2022-03-17T21:37:01.5600334Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220317191659.xml (deflated 79%) 2022-03-17T21:37:01.5601227Z adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220317191659.xml (deflated 63%) 2022-03-17T21:37:01.5602006Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220317191717.xml (deflated 54%) 2022-03-17T21:37:01.5602765Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220317191726.xml (deflated 71%) 2022-03-17T21:37:01.5603718Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220317191726.xml (deflated 69%) 2022-03-17T21:37:01.5604490Z adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220317191726.xml (deflated 66%) 2022-03-17T21:37:01.5605364Z adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220317191731.xml (deflated 71%) 2022-03-17T21:37:01.5606248Z adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220317191737.xml (deflated 54%) 2022-03-17T21:37:01.5606912Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220317191740.xml (deflated 53%) 2022-03-17T21:37:01.5607674Z adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220317191740.xml (deflated 69%) 2022-03-17T21:37:01.5608395Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220317191742.xml (deflated 81%) 2022-03-17T21:37:01.5609209Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220317191742.xml (deflated 81%) 2022-03-17T21:37:01.5610047Z adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220317191742.xml (deflated 81%) 2022-03-17T21:37:01.5610806Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220317191746.xml (deflated 57%) 2022-03-17T21:37:01.5611575Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220317191754.xml (deflated 83%) 2022-03-17T21:37:01.5612345Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220317191808.xml (deflated 80%) 2022-03-17T21:37:01.5613109Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220317191808.xml (deflated 86%) 2022-03-17T21:37:01.5613889Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20220317191851.xml (deflated 91%) 2022-03-17T21:37:01.5614602Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20220317191919.xml (deflated 72%) 2022-03-17T21:37:01.5615427Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20220317191919.xml (deflated 39%) 2022-03-17T21:37:01.5616073Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20220317191919.xml (deflated 40%) 2022-03-17T21:37:01.5616807Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20220317191919.xml (deflated 94%) 2022-03-17T21:37:01.5617605Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220317192715.xml (deflated 85%) 2022-03-17T21:37:01.5618362Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220317192748.xml (deflated 93%) 2022-03-17T21:37:01.5619067Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220317192823.xml (deflated 57%) 2022-03-17T21:37:01.5619776Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220317192831.xml (deflated 55%) 2022-03-17T21:37:01.5621028Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220317192845.xml (deflated 41%) 2022-03-17T21:37:01.5621850Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220317192851.xml (deflated 47%) 2022-03-17T21:37:01.5622669Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220317192856.xml (deflated 43%) 2022-03-17T21:37:01.5623716Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220317192856.xml (deflated 43%) 2022-03-17T21:37:01.5624548Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20220317192911.xml (deflated 51%) 2022-03-17T21:37:01.5625275Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220317192916.xml (deflated 82%) 2022-03-17T21:37:01.5626078Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220317192946.xml (deflated 90%) 2022-03-17T21:37:01.5626925Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220317192946.xml (deflated 85%) 2022-03-17T21:37:01.5627719Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220317193136.xml (deflated 41%) 2022-03-17T21:37:01.5628479Z adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220317193141.xml (deflated 41%) 2022-03-17T21:37:01.5629186Z adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220317193147.xml (deflated 68%) 2022-03-17T21:37:01.5629930Z adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220317193149.xml (deflated 79%) 2022-03-17T21:37:01.5630580Z adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220317193149.xml (deflated 86%) 2022-03-17T21:37:01.5631291Z adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220317193225.xml (deflated 63%) 2022-03-17T21:37:01.5632195Z adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220317193228.xml (deflated 90%) 2022-03-17T21:37:01.5633185Z adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220317193228.xml (deflated 73%) 2022-03-17T21:37:01.5634107Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220317193523.xml (deflated 41%) 2022-03-17T21:37:01.5635077Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220317193527.xml (deflated 41%) 2022-03-17T21:37:01.5636054Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220317193534.xml (deflated 41%) 2022-03-17T21:37:01.5636950Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220317193540.xml (deflated 41%) 2022-03-17T21:37:01.5637765Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220317193547.xml (deflated 41%) 2022-03-17T21:37:01.5638672Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220317193553.xml (deflated 41%) 2022-03-17T21:37:01.5639564Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220317193559.xml (deflated 41%) 2022-03-17T21:37:01.5640529Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220317193603.xml (deflated 41%) 2022-03-17T21:37:01.5641389Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220317193609.xml (deflated 40%) 2022-03-17T21:37:01.5642128Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193618.xml (deflated 41%) 2022-03-17T21:37:01.5643057Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193622.xml (deflated 41%) 2022-03-17T21:37:01.5643932Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193626.xml (deflated 41%) 2022-03-17T21:37:01.5644848Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193631.xml (deflated 41%) 2022-03-17T21:37:01.5645692Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193635.xml (deflated 40%) 2022-03-17T21:37:01.5646470Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193639.xml (deflated 41%) 2022-03-17T21:37:01.5647317Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193643.xml (deflated 41%) 2022-03-17T21:37:01.5648213Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220317193648.xml (deflated 41%) 2022-03-17T21:37:01.5649134Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193652.xml (deflated 42%) 2022-03-17T21:37:01.5650086Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193701.xml (deflated 43%) 2022-03-17T21:37:01.5650976Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193713.xml (deflated 42%) 2022-03-17T21:37:01.5651936Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193724.xml (deflated 44%) 2022-03-17T21:37:01.5652892Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193734.xml (deflated 44%) 2022-03-17T21:37:01.5653841Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193742.xml (deflated 44%) 2022-03-17T21:37:01.5654786Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193752.xml (deflated 43%) 2022-03-17T21:37:01.5655773Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193802.xml (deflated 43%) 2022-03-17T21:37:01.5656731Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193811.xml (deflated 43%) 2022-03-17T21:37:01.5657678Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193821.xml (deflated 44%) 2022-03-17T21:37:01.5658602Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193826.xml (deflated 43%) 2022-03-17T21:37:01.5659553Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193830.xml (deflated 43%) 2022-03-17T21:37:01.5660684Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193835.xml (deflated 43%) 2022-03-17T21:37:01.5661623Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193839.xml (deflated 43%) 2022-03-17T21:37:01.5662569Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193845.xml (deflated 43%) 2022-03-17T21:37:01.5663599Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193851.xml (deflated 43%) 2022-03-17T21:37:01.5664537Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193902.xml (deflated 43%) 2022-03-17T21:37:01.5665498Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193914.xml (deflated 43%) 2022-03-17T21:37:01.5666439Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193932.xml (deflated 43%) 2022-03-17T21:37:01.5667361Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193945.xml (deflated 43%) 2022-03-17T21:37:01.5668308Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317193957.xml (deflated 43%) 2022-03-17T21:37:01.5669245Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194002.xml (deflated 43%) 2022-03-17T21:37:01.5670191Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194009.xml (deflated 43%) 2022-03-17T21:37:01.5671166Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194017.xml (deflated 43%) 2022-03-17T21:37:01.5672089Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194024.xml (deflated 44%) 2022-03-17T21:37:01.5673009Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194034.xml (deflated 43%) 2022-03-17T21:37:01.5673934Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194044.xml (deflated 43%) 2022-03-17T21:37:01.5674905Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194054.xml (deflated 43%) 2022-03-17T21:37:01.5675877Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194104.xml (deflated 43%) 2022-03-17T21:37:01.5676825Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194114.xml (deflated 43%) 2022-03-17T21:37:01.5677762Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194124.xml (deflated 43%) 2022-03-17T21:37:01.5678694Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194134.xml (deflated 43%) 2022-03-17T21:37:01.5679711Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194144.xml (deflated 43%) 2022-03-17T21:37:01.5680884Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194154.xml (deflated 43%) 2022-03-17T21:37:01.5681826Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194204.xml (deflated 43%) 2022-03-17T21:37:01.5682759Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194213.xml (deflated 43%) 2022-03-17T21:37:01.5683759Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194223.xml (deflated 43%) 2022-03-17T21:37:01.5685088Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194233.xml (deflated 43%) 2022-03-17T21:37:01.5686194Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194243.xml (deflated 43%) 2022-03-17T21:37:01.5687265Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194253.xml (deflated 43%) 2022-03-17T21:37:01.5688201Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194303.xml (deflated 43%) 2022-03-17T21:37:01.5689205Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194313.xml (deflated 43%) 2022-03-17T21:37:01.5690071Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194320.xml (deflated 43%) 2022-03-17T21:37:01.5691028Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194330.xml (deflated 43%) 2022-03-17T21:37:01.5691950Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194338.xml (deflated 43%) 2022-03-17T21:37:01.5692950Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194346.xml (deflated 43%) 2022-03-17T21:37:01.5693919Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194356.xml (deflated 43%) 2022-03-17T21:37:01.5694764Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194406.xml (deflated 43%) 2022-03-17T21:37:01.5695711Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194410.xml (deflated 43%) 2022-03-17T21:37:01.5696750Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194414.xml (deflated 43%) 2022-03-17T21:37:01.5697740Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194419.xml (deflated 43%) 2022-03-17T21:37:01.5698626Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194423.xml (deflated 43%) 2022-03-17T21:37:01.5699566Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194430.xml (deflated 43%) 2022-03-17T21:37:01.5700878Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194437.xml (deflated 43%) 2022-03-17T21:37:01.5701862Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194443.xml (deflated 43%) 2022-03-17T21:37:01.5702814Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194450.xml (deflated 43%) 2022-03-17T21:37:01.5703749Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194456.xml (deflated 43%) 2022-03-17T21:37:01.5704796Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194503.xml (deflated 42%) 2022-03-17T21:37:01.5705768Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194509.xml (deflated 43%) 2022-03-17T21:37:01.5706685Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194519.xml (deflated 42%) 2022-03-17T21:37:01.5707642Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194529.xml (deflated 43%) 2022-03-17T21:37:01.5708589Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194533.xml (deflated 43%) 2022-03-17T21:37:01.5709508Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194543.xml (deflated 44%) 2022-03-17T21:37:01.5710450Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194548.xml (deflated 43%) 2022-03-17T21:37:01.5711389Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194552.xml (deflated 43%) 2022-03-17T21:37:01.5712320Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194557.xml (deflated 43%) 2022-03-17T21:37:01.5713260Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194604.xml (deflated 43%) 2022-03-17T21:37:01.5714281Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194609.xml (deflated 43%) 2022-03-17T21:37:01.5715186Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194613.xml (deflated 42%) 2022-03-17T21:37:01.5716149Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194622.xml (deflated 42%) 2022-03-17T21:37:01.5717119Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194633.xml (deflated 42%) 2022-03-17T21:37:01.5718046Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194644.xml (deflated 42%) 2022-03-17T21:37:01.5718995Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194653.xml (deflated 43%) 2022-03-17T21:37:01.5719929Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194711.xml (deflated 43%) 2022-03-17T21:37:01.5720876Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194730.xml (deflated 43%) 2022-03-17T21:37:01.5721818Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194748.xml (deflated 42%) 2022-03-17T21:37:01.5722747Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194807.xml (deflated 42%) 2022-03-17T21:37:01.5723804Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194825.xml (deflated 42%) 2022-03-17T21:37:01.5724707Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194841.xml (deflated 42%) 2022-03-17T21:37:01.5725738Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194858.xml (deflated 42%) 2022-03-17T21:37:01.5726693Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194914.xml (deflated 42%) 2022-03-17T21:37:01.5727645Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194930.xml (deflated 43%) 2022-03-17T21:37:01.5728569Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317194947.xml (deflated 42%) 2022-03-17T21:37:01.5729478Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317195007.xml (deflated 43%) 2022-03-17T21:37:01.5730412Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317195024.xml (deflated 42%) 2022-03-17T21:37:01.5731367Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317195044.xml (deflated 43%) 2022-03-17T21:37:01.5732270Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220317195052.xml (deflated 43%) 2022-03-17T21:37:01.5733206Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220317195102.xml (deflated 44%) 2022-03-17T21:37:01.5734266Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220317195106.xml (deflated 44%) 2022-03-17T21:37:01.5735254Z adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220317195111.xml (deflated 44%) 2022-03-17T21:37:01.5736084Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220317195118.xml (deflated 38%) 2022-03-17T21:37:01.5736883Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317195122.xml (deflated 42%) 2022-03-17T21:37:01.5737767Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317195124.xml (deflated 40%) 2022-03-17T21:37:01.5738484Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317195127.xml (deflated 40%) 2022-03-17T21:37:01.5739369Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220317195130.xml (deflated 42%) 2022-03-17T21:37:01.5740076Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317195132.xml (deflated 41%) 2022-03-17T21:37:01.5741172Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317195136.xml (deflated 41%) 2022-03-17T21:37:01.5742028Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317195142.xml (deflated 41%) 2022-03-17T21:37:01.5742773Z adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220317195146.xml (deflated 41%) 2022-03-17T21:37:01.5743521Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195152.xml (deflated 39%) 2022-03-17T21:37:01.5744187Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195156.xml (deflated 38%) 2022-03-17T21:37:01.5744830Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195201.xml (deflated 38%) 2022-03-17T21:37:01.5745637Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195205.xml (deflated 38%) 2022-03-17T21:37:01.5746241Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195210.xml (deflated 38%) 2022-03-17T21:37:01.5746873Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195215.xml (deflated 39%) 2022-03-17T21:37:01.5747593Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195219.xml (deflated 38%) 2022-03-17T21:37:01.5748236Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220317195222.xml (deflated 38%) 2022-03-17T21:37:01.5748886Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195226.xml (deflated 45%) 2022-03-17T21:37:01.5749775Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195231.xml (deflated 45%) 2022-03-17T21:37:01.5750479Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195237.xml (deflated 43%) 2022-03-17T21:37:01.5751327Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195242.xml (deflated 43%) 2022-03-17T21:37:01.5752049Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195248.xml (deflated 45%) 2022-03-17T21:37:01.5752855Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195253.xml (deflated 46%) 2022-03-17T21:37:01.5753617Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195259.xml (deflated 47%) 2022-03-17T21:37:01.5754408Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195305.xml (deflated 47%) 2022-03-17T21:37:01.5755256Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195310.xml (deflated 44%) 2022-03-17T21:37:01.5755946Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195316.xml (deflated 46%) 2022-03-17T21:37:01.5756826Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195321.xml (deflated 46%) 2022-03-17T21:37:01.5757627Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195327.xml (deflated 44%) 2022-03-17T21:37:01.5758467Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195332.xml (deflated 44%) 2022-03-17T21:37:01.5759156Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195338.xml (deflated 43%) 2022-03-17T21:37:01.5759946Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195341.xml (deflated 44%) 2022-03-17T21:37:01.5760749Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195347.xml (deflated 45%) 2022-03-17T21:37:01.5761476Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195350.xml (deflated 44%) 2022-03-17T21:37:01.5762295Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195354.xml (deflated 45%) 2022-03-17T21:37:01.5763121Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195358.xml (deflated 45%) 2022-03-17T21:37:01.5763804Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195401.xml (deflated 50%) 2022-03-17T21:37:01.5764733Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195406.xml (deflated 42%) 2022-03-17T21:37:01.5765447Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195412.xml (deflated 41%) 2022-03-17T21:37:01.5766215Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195417.xml (deflated 41%) 2022-03-17T21:37:01.5767075Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195422.xml (deflated 41%) 2022-03-17T21:37:01.5767788Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195427.xml (deflated 42%) 2022-03-17T21:37:01.5768559Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195433.xml (deflated 42%) 2022-03-17T21:37:01.5769390Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195437.xml (deflated 42%) 2022-03-17T21:37:01.5770114Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195440.xml (deflated 41%) 2022-03-17T21:37:01.5770957Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195444.xml (deflated 41%) 2022-03-17T21:37:01.5771658Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195448.xml (deflated 44%) 2022-03-17T21:37:01.5772487Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195451.xml (deflated 46%) 2022-03-17T21:37:01.5773203Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195455.xml (deflated 41%) 2022-03-17T21:37:01.5774031Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195459.xml (deflated 41%) 2022-03-17T21:37:01.5774788Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195504.xml (deflated 42%) 2022-03-17T21:37:01.5775534Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220317195508.xml (deflated 42%) 2022-03-17T21:37:01.5776363Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195512.xml (deflated 39%) 2022-03-17T21:37:01.5777114Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195515.xml (deflated 39%) 2022-03-17T21:37:01.5777838Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195521.xml (deflated 39%) 2022-03-17T21:37:01.5778602Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195525.xml (deflated 40%) 2022-03-17T21:37:01.5779356Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195528.xml (deflated 39%) 2022-03-17T21:37:01.5780054Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195532.xml (deflated 40%) 2022-03-17T21:37:01.5781138Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195537.xml (deflated 39%) 2022-03-17T21:37:01.5781793Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195544.xml (deflated 39%) 2022-03-17T21:37:01.5782607Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195548.xml (deflated 40%) 2022-03-17T21:37:01.5783268Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195553.xml (deflated 39%) 2022-03-17T21:37:01.5784081Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195559.xml (deflated 39%) 2022-03-17T21:37:01.5784927Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195603.xml (deflated 39%) 2022-03-17T21:37:01.5785555Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195606.xml (deflated 39%) 2022-03-17T21:37:01.5786343Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195610.xml (deflated 40%) 2022-03-17T21:37:01.5787033Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195614.xml (deflated 39%) 2022-03-17T21:37:01.5787803Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195618.xml (deflated 39%) 2022-03-17T21:37:01.5788492Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195624.xml (deflated 40%) 2022-03-17T21:37:01.5789242Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195628.xml (deflated 40%) 2022-03-17T21:37:01.5789994Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195632.xml (deflated 40%) 2022-03-17T21:37:01.5790705Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195638.xml (deflated 40%) 2022-03-17T21:37:01.5791501Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195641.xml (deflated 40%) 2022-03-17T21:37:01.5792144Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195645.xml (deflated 40%) 2022-03-17T21:37:01.5792919Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195651.xml (deflated 40%) 2022-03-17T21:37:01.5793615Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195655.xml (deflated 40%) 2022-03-17T21:37:01.5794345Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195658.xml (deflated 40%) 2022-03-17T21:37:01.5795115Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195704.xml (deflated 40%) 2022-03-17T21:37:01.5795895Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195708.xml (deflated 40%) 2022-03-17T21:37:01.5796627Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195712.xml (deflated 40%) 2022-03-17T21:37:01.5797371Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195717.xml (deflated 39%) 2022-03-17T21:37:01.5798086Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195721.xml (deflated 40%) 2022-03-17T21:37:01.5798846Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195726.xml (deflated 39%) 2022-03-17T21:37:01.5799557Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195733.xml (deflated 39%) 2022-03-17T21:37:01.5800320Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195737.xml (deflated 39%) 2022-03-17T21:37:01.5801009Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195741.xml (deflated 40%) 2022-03-17T21:37:01.5801736Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195747.xml (deflated 39%) 2022-03-17T21:37:01.5802516Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195750.xml (deflated 40%) 2022-03-17T21:37:01.5803334Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195754.xml (deflated 40%) 2022-03-17T21:37:01.5804002Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195801.xml (deflated 40%) 2022-03-17T21:37:01.5804782Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195805.xml (deflated 39%) 2022-03-17T21:37:01.5805458Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195809.xml (deflated 40%) 2022-03-17T21:37:01.5806252Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195813.xml (deflated 40%) 2022-03-17T21:37:01.5806907Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195818.xml (deflated 40%) 2022-03-17T21:37:01.5807645Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195822.xml (deflated 40%) 2022-03-17T21:37:01.5808432Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195826.xml (deflated 41%) 2022-03-17T21:37:01.5809094Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195827.xml (deflated 40%) 2022-03-17T21:37:01.5809821Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195831.xml (deflated 42%) 2022-03-17T21:37:01.5810584Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195832.xml (deflated 40%) 2022-03-17T21:37:01.5811266Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220317195838.xml (deflated 39%) 2022-03-17T21:37:01.5812078Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195842.xml (deflated 38%) 2022-03-17T21:37:01.5812656Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195843.xml (deflated 39%) 2022-03-17T21:37:01.5813380Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195844.xml (deflated 39%) 2022-03-17T21:37:01.5814068Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195845.xml (deflated 38%) 2022-03-17T21:37:01.5814715Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220317195846.xml (deflated 40%) 2022-03-17T21:37:01.5815462Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220317195847.xml (deflated 39%) 2022-03-17T21:37:01.5816094Z adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220317195850.xml (deflated 40%) 2022-03-17T21:37:01.5816795Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195856.xml (deflated 39%) 2022-03-17T21:37:01.5817455Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195902.xml (deflated 39%) 2022-03-17T21:37:01.5818059Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195909.xml (deflated 40%) 2022-03-17T21:37:01.5818701Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195912.xml (deflated 38%) 2022-03-17T21:37:01.5819455Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195917.xml (deflated 38%) 2022-03-17T21:37:01.5820106Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195921.xml (deflated 40%) 2022-03-17T21:37:01.5821021Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195925.xml (deflated 40%) 2022-03-17T21:37:01.5821684Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195928.xml (deflated 39%) 2022-03-17T21:37:01.5822367Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195932.xml (deflated 37%) 2022-03-17T21:37:01.5823046Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195937.xml (deflated 38%) 2022-03-17T21:37:01.5823785Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195942.xml (deflated 38%) 2022-03-17T21:37:01.5824351Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195948.xml (deflated 38%) 2022-03-17T21:37:01.5824979Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195954.xml (deflated 38%) 2022-03-17T21:37:01.5825617Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317195959.xml (deflated 40%) 2022-03-17T21:37:01.5826312Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317200003.xml (deflated 39%) 2022-03-17T21:37:01.5826915Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220317200008.xml (deflated 38%) 2022-03-17T21:37:01.5827653Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200013.xml (deflated 42%) 2022-03-17T21:37:01.5828416Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200019.xml (deflated 42%) 2022-03-17T21:37:01.5829222Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200026.xml (deflated 41%) 2022-03-17T21:37:01.5829985Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200032.xml (deflated 42%) 2022-03-17T21:37:01.5830777Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200039.xml (deflated 41%) 2022-03-17T21:37:01.5831610Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200045.xml (deflated 42%) 2022-03-17T21:37:01.5832355Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200051.xml (deflated 42%) 2022-03-17T21:37:01.5833210Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200057.xml (deflated 42%) 2022-03-17T21:37:01.5834021Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200103.xml (deflated 45%) 2022-03-17T21:37:01.5834767Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200109.xml (deflated 45%) 2022-03-17T21:37:01.5835598Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200114.xml (deflated 43%) 2022-03-17T21:37:01.5836328Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200120.xml (deflated 43%) 2022-03-17T21:37:01.5837113Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200126.xml (deflated 46%) 2022-03-17T21:37:01.5837916Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200131.xml (deflated 46%) 2022-03-17T21:37:01.5838740Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200137.xml (deflated 47%) 2022-03-17T21:37:01.5839491Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200142.xml (deflated 47%) 2022-03-17T21:37:01.5840268Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200148.xml (deflated 44%) 2022-03-17T21:37:01.5841070Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200153.xml (deflated 46%) 2022-03-17T21:37:01.5841812Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200159.xml (deflated 46%) 2022-03-17T21:37:01.5842608Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200204.xml (deflated 44%) 2022-03-17T21:37:01.5843467Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200210.xml (deflated 44%) 2022-03-17T21:37:01.5844146Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200215.xml (deflated 42%) 2022-03-17T21:37:01.5844946Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200222.xml (deflated 42%) 2022-03-17T21:37:01.5845753Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200228.xml (deflated 42%) 2022-03-17T21:37:01.5846523Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200234.xml (deflated 45%) 2022-03-17T21:37:01.5847281Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200240.xml (deflated 44%) 2022-03-17T21:37:01.5848064Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200246.xml (deflated 43%) 2022-03-17T21:37:01.5848916Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200250.xml (deflated 42%) 2022-03-17T21:37:01.5849605Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200256.xml (deflated 41%) 2022-03-17T21:37:01.5850445Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200300.xml (deflated 42%) 2022-03-17T21:37:01.5851160Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200306.xml (deflated 42%) 2022-03-17T21:37:01.5851998Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200312.xml (deflated 42%) 2022-03-17T21:37:01.5852754Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200319.xml (deflated 41%) 2022-03-17T21:37:01.5853547Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200324.xml (deflated 41%) 2022-03-17T21:37:01.5854375Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200330.xml (deflated 41%) 2022-03-17T21:37:01.5855117Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200335.xml (deflated 41%) 2022-03-17T21:37:01.5855890Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200341.xml (deflated 41%) 2022-03-17T21:37:01.5856729Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200346.xml (deflated 41%) 2022-03-17T21:37:01.5857432Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200352.xml (deflated 42%) 2022-03-17T21:37:01.5858284Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200358.xml (deflated 41%) 2022-03-17T21:37:01.5858976Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200404.xml (deflated 42%) 2022-03-17T21:37:01.5859802Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200411.xml (deflated 41%) 2022-03-17T21:37:01.5860794Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200417.xml (deflated 41%) 2022-03-17T21:37:01.5861558Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200425.xml (deflated 43%) 2022-03-17T21:37:01.5862383Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200428.xml (deflated 41%) 2022-03-17T21:37:01.5863088Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200432.xml (deflated 42%) 2022-03-17T21:37:01.5863876Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200438.xml (deflated 42%) 2022-03-17T21:37:01.5864717Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200445.xml (deflated 42%) 2022-03-17T21:37:01.5865440Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200451.xml (deflated 42%) 2022-03-17T21:37:01.5866198Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200458.xml (deflated 42%) 2022-03-17T21:37:01.5867013Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200501.xml (deflated 42%) 2022-03-17T21:37:01.5867755Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200505.xml (deflated 42%) 2022-03-17T21:37:01.5868532Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200510.xml (deflated 43%) 2022-03-17T21:37:01.5869397Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200514.xml (deflated 43%) 2022-03-17T21:37:01.5870160Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200520.xml (deflated 42%) 2022-03-17T21:37:01.5870848Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200527.xml (deflated 42%) 2022-03-17T21:37:01.5871782Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200549.xml (deflated 44%) 2022-03-17T21:37:01.5872500Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200555.xml (deflated 42%) 2022-03-17T21:37:01.5873275Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200601.xml (deflated 42%) 2022-03-17T21:37:01.5874056Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200605.xml (deflated 41%) 2022-03-17T21:37:01.5874836Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220317200611.xml (deflated 41%) 2022-03-17T21:37:01.5875711Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200617.xml (deflated 42%) 2022-03-17T21:37:01.5876403Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200621.xml (deflated 42%) 2022-03-17T21:37:01.5877207Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200625.xml (deflated 42%) 2022-03-17T21:37:01.5877872Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200626.xml (deflated 42%) 2022-03-17T21:37:01.5878704Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200629.xml (deflated 41%) 2022-03-17T21:37:01.5879414Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200633.xml (deflated 41%) 2022-03-17T21:37:01.5880180Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200637.xml (deflated 41%) 2022-03-17T21:37:01.5881003Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200640.xml (deflated 43%) 2022-03-17T21:37:01.5881650Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220317200641.xml (deflated 42%) 2022-03-17T21:37:01.5882482Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220317200645.xml (deflated 42%) 2022-03-17T21:37:01.5883179Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200646.xml (deflated 38%) 2022-03-17T21:37:01.5883971Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200651.xml (deflated 39%) 2022-03-17T21:37:01.5884639Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200657.xml (deflated 39%) 2022-03-17T21:37:01.5885402Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200703.xml (deflated 39%) 2022-03-17T21:37:01.5886105Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200710.xml (deflated 39%) 2022-03-17T21:37:01.5886855Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200716.xml (deflated 39%) 2022-03-17T21:37:01.5887669Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200722.xml (deflated 39%) 2022-03-17T21:37:01.5888329Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200728.xml (deflated 39%) 2022-03-17T21:37:01.5889147Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200733.xml (deflated 39%) 2022-03-17T21:37:01.5889777Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200739.xml (deflated 39%) 2022-03-17T21:37:01.5890658Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200746.xml (deflated 39%) 2022-03-17T21:37:01.5891333Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200752.xml (deflated 39%) 2022-03-17T21:37:01.5892138Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200757.xml (deflated 39%) 2022-03-17T21:37:01.5892783Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200803.xml (deflated 39%) 2022-03-17T21:37:01.5893548Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200810.xml (deflated 39%) 2022-03-17T21:37:01.5894230Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200815.xml (deflated 39%) 2022-03-17T21:37:01.5894986Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220317200821.xml (deflated 39%) 2022-03-17T21:37:01.5895791Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220317200827.xml (deflated 40%) 2022-03-17T21:37:01.5896415Z adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220317200830.xml (deflated 40%) 2022-03-17T21:37:01.5897301Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200840.xml (deflated 44%) 2022-03-17T21:37:01.5898265Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200842.xml (deflated 44%) 2022-03-17T21:37:01.5899147Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220317200845.xml (deflated 43%) 2022-03-17T21:37:01.5900021Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200849.xml (deflated 41%) 2022-03-17T21:37:01.5901115Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200854.xml (deflated 41%) 2022-03-17T21:37:01.5901924Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200902.xml (deflated 41%) 2022-03-17T21:37:01.5902672Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220317200910.xml (deflated 41%) 2022-03-17T21:37:01.5903505Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200917.xml (deflated 42%) 2022-03-17T21:37:01.5904328Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200924.xml (deflated 42%) 2022-03-17T21:37:01.5905167Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200930.xml (deflated 42%) 2022-03-17T21:37:01.5906007Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200936.xml (deflated 42%) 2022-03-17T21:37:01.5906807Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200943.xml (deflated 42%) 2022-03-17T21:37:01.5907663Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200949.xml (deflated 42%) 2022-03-17T21:37:01.5908549Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317200956.xml (deflated 42%) 2022-03-17T21:37:01.5909289Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220317201002.xml (deflated 42%) 2022-03-17T21:37:01.5910219Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317201011.xml (deflated 41%) 2022-03-17T21:37:01.5911034Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317201019.xml (deflated 42%) 2022-03-17T21:37:01.5911928Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317201027.xml (deflated 41%) 2022-03-17T21:37:01.5912650Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220317201034.xml (deflated 41%) 2022-03-17T21:37:01.5913477Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201042.xml (deflated 42%) 2022-03-17T21:37:01.5914401Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201048.xml (deflated 42%) 2022-03-17T21:37:01.5915150Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201055.xml (deflated 42%) 2022-03-17T21:37:01.5915959Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201101.xml (deflated 42%) 2022-03-17T21:37:01.5916811Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201108.xml (deflated 42%) 2022-03-17T21:37:01.5917760Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201114.xml (deflated 42%) 2022-03-17T21:37:01.5918618Z adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220317201121.xml (deflated 42%) 2022-03-17T21:37:01.5919328Z adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220317201127.xml (deflated 83%) 2022-03-17T21:37:01.5920169Z adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20220317201127.xml (deflated 85%) 2022-03-17T21:37:01.5921007Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201140.xml (deflated 41%) 2022-03-17T21:37:01.5921709Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201146.xml (deflated 42%) 2022-03-17T21:37:01.5922538Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201147.xml (deflated 42%) 2022-03-17T21:37:01.5923311Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201151.xml (deflated 42%) 2022-03-17T21:37:01.5924142Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201200.xml (deflated 43%) 2022-03-17T21:37:01.5924847Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201201.xml (deflated 42%) 2022-03-17T21:37:01.5925712Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201202.xml (deflated 41%) 2022-03-17T21:37:01.5926393Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201210.xml (deflated 40%) 2022-03-17T21:37:01.5927197Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201217.xml (deflated 40%) 2022-03-17T21:37:01.5927917Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201223.xml (deflated 40%) 2022-03-17T21:37:01.5928740Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201230.xml (deflated 40%) 2022-03-17T21:37:01.5929494Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201238.xml (deflated 40%) 2022-03-17T21:37:01.5930271Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201245.xml (deflated 40%) 2022-03-17T21:37:01.5931056Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201251.xml (deflated 41%) 2022-03-17T21:37:01.5931788Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201259.xml (deflated 42%) 2022-03-17T21:37:01.5932640Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201303.xml (deflated 41%) 2022-03-17T21:37:01.5933302Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201309.xml (deflated 42%) 2022-03-17T21:37:01.5934142Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201316.xml (deflated 42%) 2022-03-17T21:37:01.5934827Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201317.xml (deflated 46%) 2022-03-17T21:37:01.5935656Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201324.xml (deflated 47%) 2022-03-17T21:37:01.5936351Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201331.xml (deflated 48%) 2022-03-17T21:37:01.5937167Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201338.xml (deflated 46%) 2022-03-17T21:37:01.5937916Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201345.xml (deflated 42%) 2022-03-17T21:37:01.5938699Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201347.xml (deflated 44%) 2022-03-17T21:37:01.5939556Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201348.xml (deflated 44%) 2022-03-17T21:37:01.5940377Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201349.xml (deflated 44%) 2022-03-17T21:37:01.5941243Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201350.xml (deflated 44%) 2022-03-17T21:37:01.5941934Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201351.xml (deflated 44%) 2022-03-17T21:37:01.5942676Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201352.xml (deflated 42%) 2022-03-17T21:37:01.5943501Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201353.xml (deflated 40%) 2022-03-17T21:37:01.5944197Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201401.xml (deflated 40%) 2022-03-17T21:37:01.5944979Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201409.xml (deflated 42%) 2022-03-17T21:37:01.5945716Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201411.xml (deflated 42%) 2022-03-17T21:37:01.5946470Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201412.xml (deflated 40%) 2022-03-17T21:37:01.5947229Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201418.xml (deflated 40%) 2022-03-17T21:37:01.5948002Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201424.xml (deflated 40%) 2022-03-17T21:37:01.5948926Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201431.xml (deflated 40%) 2022-03-17T21:37:01.5949599Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201439.xml (deflated 41%) 2022-03-17T21:37:01.5950386Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201440.xml (deflated 41%) 2022-03-17T21:37:01.5951112Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201441.xml (deflated 42%) 2022-03-17T21:37:01.5951939Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201442.xml (deflated 41%) 2022-03-17T21:37:01.5952658Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201443.xml (deflated 41%) 2022-03-17T21:37:01.5953549Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201445.xml (deflated 41%) 2022-03-17T21:37:01.5954362Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201446.xml (deflated 41%) 2022-03-17T21:37:01.5955077Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201447.xml (deflated 41%) 2022-03-17T21:37:01.5955958Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201448.xml (deflated 41%) 2022-03-17T21:37:01.5956672Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201449.xml (deflated 42%) 2022-03-17T21:37:01.5957465Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201450.xml (deflated 42%) 2022-03-17T21:37:01.5958189Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201451.xml (deflated 41%) 2022-03-17T21:37:01.5958942Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201453.xml (deflated 42%) 2022-03-17T21:37:01.5959774Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201454.xml (deflated 42%) 2022-03-17T21:37:01.5960450Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201455.xml (deflated 41%) 2022-03-17T21:37:01.5961308Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201456.xml (deflated 41%) 2022-03-17T21:37:01.5961965Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201457.xml (deflated 41%) 2022-03-17T21:37:01.5962816Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201458.xml (deflated 41%) 2022-03-17T21:37:01.5963493Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201459.xml (deflated 41%) 2022-03-17T21:37:01.5964289Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201501.xml (deflated 42%) 2022-03-17T21:37:01.5964998Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201502.xml (deflated 42%) 2022-03-17T21:37:01.5965857Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201503.xml (deflated 42%) 2022-03-17T21:37:01.5966600Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201504.xml (deflated 42%) 2022-03-17T21:37:01.5967277Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201505.xml (deflated 42%) 2022-03-17T21:37:01.5968170Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201506.xml (deflated 43%) 2022-03-17T21:37:01.5968844Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201508.xml (deflated 43%) 2022-03-17T21:37:01.5969620Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201509.xml (deflated 42%) 2022-03-17T21:37:01.5970350Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201510.xml (deflated 41%) 2022-03-17T21:37:01.5971189Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201517.xml (deflated 42%) 2022-03-17T21:37:01.5971889Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201518.xml (deflated 42%) 2022-03-17T21:37:01.5972673Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201519.xml (deflated 41%) 2022-03-17T21:37:01.5973395Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201520.xml (deflated 41%) 2022-03-17T21:37:01.5974172Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201526.xml (deflated 40%) 2022-03-17T21:37:01.5975047Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201532.xml (deflated 40%) 2022-03-17T21:37:01.5975745Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201538.xml (deflated 43%) 2022-03-17T21:37:01.5976536Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201540.xml (deflated 43%) 2022-03-17T21:37:01.5977266Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201541.xml (deflated 41%) 2022-03-17T21:37:01.5978114Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201547.xml (deflated 41%) 2022-03-17T21:37:01.5978786Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201554.xml (deflated 43%) 2022-03-17T21:37:01.5979573Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201555.xml (deflated 41%) 2022-03-17T21:37:01.5980485Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201602.xml (deflated 43%) 2022-03-17T21:37:01.5981249Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201603.xml (deflated 42%) 2022-03-17T21:37:01.5981998Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201607.xml (deflated 43%) 2022-03-17T21:37:01.5982838Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201608.xml (deflated 42%) 2022-03-17T21:37:01.5983504Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201610.xml (deflated 41%) 2022-03-17T21:37:01.5984324Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201618.xml (deflated 41%) 2022-03-17T21:37:01.5985030Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201626.xml (deflated 43%) 2022-03-17T21:37:01.5985763Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201627.xml (deflated 41%) 2022-03-17T21:37:01.5986639Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201635.xml (deflated 42%) 2022-03-17T21:37:01.5987440Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201636.xml (deflated 42%) 2022-03-17T21:37:01.5988135Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201640.xml (deflated 43%) 2022-03-17T21:37:01.5988920Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201642.xml (deflated 42%) 2022-03-17T21:37:01.5989670Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201643.xml (deflated 41%) 2022-03-17T21:37:01.5990424Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201649.xml (deflated 41%) 2022-03-17T21:37:01.5991218Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201656.xml (deflated 42%) 2022-03-17T21:37:01.5991946Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201657.xml (deflated 41%) 2022-03-17T21:37:01.5992770Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201704.xml (deflated 43%) 2022-03-17T21:37:01.5993453Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201705.xml (deflated 42%) 2022-03-17T21:37:01.5994301Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201709.xml (deflated 41%) 2022-03-17T21:37:01.5995035Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201716.xml (deflated 42%) 2022-03-17T21:37:01.5995811Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201720.xml (deflated 43%) 2022-03-17T21:37:01.5996638Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201721.xml (deflated 43%) 2022-03-17T21:37:01.5997315Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201723.xml (deflated 41%) 2022-03-17T21:37:01.5998129Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201730.xml (deflated 43%) 2022-03-17T21:37:01.5998916Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201731.xml (deflated 42%) 2022-03-17T21:37:01.5999602Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201735.xml (deflated 43%) 2022-03-17T21:37:01.6000438Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201736.xml (deflated 42%) 2022-03-17T21:37:01.6001108Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201741.xml (deflated 43%) 2022-03-17T21:37:01.6001943Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201742.xml (deflated 43%) 2022-03-17T21:37:01.6002635Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201743.xml (deflated 44%) 2022-03-17T21:37:01.6003451Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201744.xml (deflated 41%) 2022-03-17T21:37:01.6004145Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201745.xml (deflated 41%) 2022-03-17T21:37:01.6004962Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201746.xml (deflated 41%) 2022-03-17T21:37:01.6005714Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201751.xml (deflated 41%) 2022-03-17T21:37:01.6006567Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201756.xml (deflated 42%) 2022-03-17T21:37:01.6007232Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201801.xml (deflated 41%) 2022-03-17T21:37:01.6007986Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201806.xml (deflated 41%) 2022-03-17T21:37:01.6008763Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201810.xml (deflated 40%) 2022-03-17T21:37:01.6009509Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201816.xml (deflated 41%) 2022-03-17T21:37:01.6010285Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201820.xml (deflated 42%) 2022-03-17T21:37:01.6011085Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201821.xml (deflated 41%) 2022-03-17T21:37:01.6011793Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201828.xml (deflated 42%) 2022-03-17T21:37:01.6012652Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201829.xml (deflated 42%) 2022-03-17T21:37:01.6013366Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201830.xml (deflated 44%) 2022-03-17T21:37:01.6014147Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201831.xml (deflated 40%) 2022-03-17T21:37:01.6014955Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201838.xml (deflated 41%) 2022-03-17T21:37:01.6015653Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201844.xml (deflated 41%) 2022-03-17T21:37:01.6016457Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201849.xml (deflated 40%) 2022-03-17T21:37:01.6017142Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201855.xml (deflated 40%) 2022-03-17T21:37:01.6017903Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201901.xml (deflated 40%) 2022-03-17T21:37:01.6018758Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201907.xml (deflated 41%) 2022-03-17T21:37:01.6019425Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201913.xml (deflated 41%) 2022-03-17T21:37:01.6020193Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201920.xml (deflated 41%) 2022-03-17T21:37:01.6021132Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201926.xml (deflated 40%) 2022-03-17T21:37:01.6021883Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201933.xml (deflated 40%) 2022-03-17T21:37:01.6022635Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201939.xml (deflated 41%) 2022-03-17T21:37:01.6023505Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201945.xml (deflated 43%) 2022-03-17T21:37:01.6024219Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201947.xml (deflated 41%) 2022-03-17T21:37:01.6025056Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317201954.xml (deflated 40%) 2022-03-17T21:37:01.6025855Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202001.xml (deflated 41%) 2022-03-17T21:37:01.6026576Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202009.xml (deflated 41%) 2022-03-17T21:37:01.6027410Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202016.xml (deflated 40%) 2022-03-17T21:37:01.6028094Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202024.xml (deflated 41%) 2022-03-17T21:37:01.6028867Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202031.xml (deflated 41%) 2022-03-17T21:37:01.6029687Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202038.xml (deflated 40%) 2022-03-17T21:37:01.6030375Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202046.xml (deflated 42%) 2022-03-17T21:37:01.6031140Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202047.xml (deflated 42%) 2022-03-17T21:37:01.6031948Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202048.xml (deflated 42%) 2022-03-17T21:37:01.6032746Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202049.xml (deflated 42%) 2022-03-17T21:37:01.6033478Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202050.xml (deflated 42%) 2022-03-17T21:37:01.6034259Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202052.xml (deflated 42%) 2022-03-17T21:37:01.6035113Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202053.xml (deflated 42%) 2022-03-17T21:37:01.6035776Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202054.xml (deflated 42%) 2022-03-17T21:37:01.6036603Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202055.xml (deflated 42%) 2022-03-17T21:37:01.6037298Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202056.xml (deflated 42%) 2022-03-17T21:37:01.6038122Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202057.xml (deflated 42%) 2022-03-17T21:37:01.6038810Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202058.xml (deflated 42%) 2022-03-17T21:37:01.6039585Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202100.xml (deflated 41%) 2022-03-17T21:37:01.6040316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202107.xml (deflated 41%) 2022-03-17T21:37:01.6041094Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202113.xml (deflated 41%) 2022-03-17T21:37:01.6041907Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202120.xml (deflated 42%) 2022-03-17T21:37:01.6042607Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202121.xml (deflated 41%) 2022-03-17T21:37:01.6043380Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202129.xml (deflated 41%) 2022-03-17T21:37:01.6044175Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202134.xml (deflated 41%) 2022-03-17T21:37:01.6044954Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202151.xml (deflated 41%) 2022-03-17T21:37:01.6045779Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202159.xml (deflated 41%) 2022-03-17T21:37:01.6046475Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202206.xml (deflated 42%) 2022-03-17T21:37:01.6047291Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202213.xml (deflated 41%) 2022-03-17T21:37:01.6048009Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202220.xml (deflated 41%) 2022-03-17T21:37:01.6048846Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202227.xml (deflated 42%) 2022-03-17T21:37:01.6049527Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202233.xml (deflated 42%) 2022-03-17T21:37:01.6050300Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202240.xml (deflated 41%) 2022-03-17T21:37:01.6051156Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202247.xml (deflated 41%) 2022-03-17T21:37:01.6051854Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202253.xml (deflated 41%) 2022-03-17T21:37:01.6052687Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202259.xml (deflated 41%) 2022-03-17T21:37:01.6053380Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202305.xml (deflated 41%) 2022-03-17T21:37:01.6054228Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202311.xml (deflated 40%) 2022-03-17T21:37:01.6054882Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202318.xml (deflated 41%) 2022-03-17T21:37:01.6055662Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202325.xml (deflated 41%) 2022-03-17T21:37:01.6056407Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202332.xml (deflated 41%) 2022-03-17T21:37:01.6057205Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202536.xml (deflated 40%) 2022-03-17T21:37:01.6057922Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202542.xml (deflated 41%) 2022-03-17T21:37:01.6058696Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202549.xml (deflated 41%) 2022-03-17T21:37:01.6059439Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202553.xml (deflated 41%) 2022-03-17T21:37:01.6060196Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202557.xml (deflated 41%) 2022-03-17T21:37:01.6061158Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202603.xml (deflated 41%) 2022-03-17T21:37:01.6061988Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202609.xml (deflated 41%) 2022-03-17T21:37:01.6062749Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202614.xml (deflated 42%) 2022-03-17T21:37:01.6063588Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202615.xml (deflated 42%) 2022-03-17T21:37:01.6064269Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202616.xml (deflated 41%) 2022-03-17T21:37:01.6065077Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202624.xml (deflated 41%) 2022-03-17T21:37:01.6065780Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202625.xml (deflated 42%) 2022-03-17T21:37:01.6066551Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202626.xml (deflated 41%) 2022-03-17T21:37:01.6067408Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202633.xml (deflated 41%) 2022-03-17T21:37:01.6068073Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202640.xml (deflated 42%) 2022-03-17T21:37:01.6068883Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202644.xml (deflated 40%) 2022-03-17T21:37:01.6069604Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202651.xml (deflated 41%) 2022-03-17T21:37:01.6070426Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202655.xml (deflated 40%) 2022-03-17T21:37:01.6071262Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202700.xml (deflated 41%) 2022-03-17T21:37:01.6071951Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202704.xml (deflated 41%) 2022-03-17T21:37:01.6072788Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202710.xml (deflated 43%) 2022-03-17T21:37:01.6073465Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202711.xml (deflated 43%) 2022-03-17T21:37:01.6074267Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202712.xml (deflated 43%) 2022-03-17T21:37:01.6074965Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202713.xml (deflated 43%) 2022-03-17T21:37:01.6075791Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202715.xml (deflated 41%) 2022-03-17T21:37:01.6076495Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202737.xml (deflated 41%) 2022-03-17T21:37:01.6077306Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202759.xml (deflated 42%) 2022-03-17T21:37:01.6078011Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202800.xml (deflated 42%) 2022-03-17T21:37:01.6078742Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202801.xml (deflated 42%) 2022-03-17T21:37:01.6079510Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202802.xml (deflated 42%) 2022-03-17T21:37:01.6080304Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202803.xml (deflated 42%) 2022-03-17T21:37:01.6081016Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202805.xml (deflated 40%) 2022-03-17T21:37:01.6081889Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202810.xml (deflated 41%) 2022-03-17T21:37:01.6082599Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202816.xml (deflated 41%) 2022-03-17T21:37:01.6083362Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202823.xml (deflated 41%) 2022-03-17T21:37:01.6084108Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202828.xml (deflated 40%) 2022-03-17T21:37:01.6084889Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202836.xml (deflated 42%) 2022-03-17T21:37:01.6085619Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202837.xml (deflated 42%) 2022-03-17T21:37:01.6086395Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202838.xml (deflated 42%) 2022-03-17T21:37:01.6087204Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202839.xml (deflated 40%) 2022-03-17T21:37:01.6087911Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202843.xml (deflated 41%) 2022-03-17T21:37:01.6088744Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202848.xml (deflated 42%) 2022-03-17T21:37:01.6089502Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202849.xml (deflated 43%) 2022-03-17T21:37:01.6090270Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202850.xml (deflated 41%) 2022-03-17T21:37:01.6091101Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202857.xml (deflated 41%) 2022-03-17T21:37:01.6091764Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202904.xml (deflated 41%) 2022-03-17T21:37:01.6092606Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202911.xml (deflated 40%) 2022-03-17T21:37:01.6093283Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202917.xml (deflated 42%) 2022-03-17T21:37:01.6094046Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202918.xml (deflated 42%) 2022-03-17T21:37:01.6094802Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202919.xml (deflated 42%) 2022-03-17T21:37:01.6095655Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202921.xml (deflated 42%) 2022-03-17T21:37:01.6096312Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202922.xml (deflated 42%) 2022-03-17T21:37:01.6097087Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202923.xml (deflated 42%) 2022-03-17T21:37:01.6097926Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202924.xml (deflated 42%) 2022-03-17T21:37:01.6098594Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202925.xml (deflated 42%) 2022-03-17T21:37:01.6099361Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202926.xml (deflated 42%) 2022-03-17T21:37:01.6100207Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202927.xml (deflated 42%) 2022-03-17T21:37:01.6101129Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202928.xml (deflated 41%) 2022-03-17T21:37:01.6101900Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202935.xml (deflated 42%) 2022-03-17T21:37:01.6102709Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202936.xml (deflated 42%) 2022-03-17T21:37:01.6103413Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202937.xml (deflated 41%) 2022-03-17T21:37:01.6104258Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202943.xml (deflated 41%) 2022-03-17T21:37:01.6104912Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202949.xml (deflated 42%) 2022-03-17T21:37:01.6105670Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202951.xml (deflated 42%) 2022-03-17T21:37:01.6106453Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202952.xml (deflated 42%) 2022-03-17T21:37:01.6107183Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202953.xml (deflated 42%) 2022-03-17T21:37:01.6108036Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317202954.xml (deflated 41%) 2022-03-17T21:37:01.6108836Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203002.xml (deflated 41%) 2022-03-17T21:37:01.6109559Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203010.xml (deflated 42%) 2022-03-17T21:37:01.6110386Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203011.xml (deflated 42%) 2022-03-17T21:37:01.6111086Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203012.xml (deflated 42%) 2022-03-17T21:37:01.6111887Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203014.xml (deflated 44%) 2022-03-17T21:37:01.6112598Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203015.xml (deflated 43%) 2022-03-17T21:37:01.6113446Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203016.xml (deflated 43%) 2022-03-17T21:37:01.6114114Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203017.xml (deflated 43%) 2022-03-17T21:37:01.6114902Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203018.xml (deflated 43%) 2022-03-17T21:37:01.6115631Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203019.xml (deflated 41%) 2022-03-17T21:37:01.6116415Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203025.xml (deflated 41%) 2022-03-17T21:37:01.6117160Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203031.xml (deflated 41%) 2022-03-17T21:37:01.6117972Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203037.xml (deflated 44%) 2022-03-17T21:37:01.6118679Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203039.xml (deflated 43%) 2022-03-17T21:37:01.6119506Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203040.xml (deflated 43%) 2022-03-17T21:37:01.6120248Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203041.xml (deflated 43%) 2022-03-17T21:37:01.6121021Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203042.xml (deflated 43%) 2022-03-17T21:37:01.6121851Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203043.xml (deflated 43%) 2022-03-17T21:37:01.6122523Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203044.xml (deflated 41%) 2022-03-17T21:37:01.6123348Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203051.xml (deflated 41%) 2022-03-17T21:37:01.6124108Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203053.xml (deflated 40%) 2022-03-17T21:37:01.6124908Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203059.xml (deflated 41%) 2022-03-17T21:37:01.6125619Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203106.xml (deflated 41%) 2022-03-17T21:37:01.6126379Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203118.xml (deflated 41%) 2022-03-17T21:37:01.6127169Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203130.xml (deflated 42%) 2022-03-17T21:37:01.6128002Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203137.xml (deflated 42%) 2022-03-17T21:37:01.6128689Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203138.xml (deflated 42%) 2022-03-17T21:37:01.6129478Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203142.xml (deflated 42%) 2022-03-17T21:37:01.6130213Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203152.xml (deflated 42%) 2022-03-17T21:37:01.6130968Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203153.xml (deflated 42%) 2022-03-17T21:37:01.6131814Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203154.xml (deflated 41%) 2022-03-17T21:37:01.6132493Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203202.xml (deflated 40%) 2022-03-17T21:37:01.6133297Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203208.xml (deflated 40%) 2022-03-17T21:37:01.6134005Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203215.xml (deflated 40%) 2022-03-17T21:37:01.6134768Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203222.xml (deflated 40%) 2022-03-17T21:37:01.6135536Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203230.xml (deflated 40%) 2022-03-17T21:37:01.6136272Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203236.xml (deflated 40%) 2022-03-17T21:37:01.6137035Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203243.xml (deflated 41%) 2022-03-17T21:37:01.6137770Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203251.xml (deflated 41%) 2022-03-17T21:37:01.6138608Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203255.xml (deflated 41%) 2022-03-17T21:37:01.6139372Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203301.xml (deflated 42%) 2022-03-17T21:37:01.6140113Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203308.xml (deflated 42%) 2022-03-17T21:37:01.6141050Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203309.xml (deflated 46%) 2022-03-17T21:37:01.6141786Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203316.xml (deflated 47%) 2022-03-17T21:37:01.6142632Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203323.xml (deflated 48%) 2022-03-17T21:37:01.6143323Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203330.xml (deflated 46%) 2022-03-17T21:37:01.6144142Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203337.xml (deflated 42%) 2022-03-17T21:37:01.6144822Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203338.xml (deflated 44%) 2022-03-17T21:37:01.6145658Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203339.xml (deflated 44%) 2022-03-17T21:37:01.6146502Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203341.xml (deflated 44%) 2022-03-17T21:37:01.6147170Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203342.xml (deflated 44%) 2022-03-17T21:37:01.6148002Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203343.xml (deflated 44%) 2022-03-17T21:37:01.6148688Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203344.xml (deflated 41%) 2022-03-17T21:37:01.6149490Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203345.xml (deflated 40%) 2022-03-17T21:37:01.6150195Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203353.xml (deflated 40%) 2022-03-17T21:37:01.6150959Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203401.xml (deflated 42%) 2022-03-17T21:37:01.6151814Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203402.xml (deflated 42%) 2022-03-17T21:37:01.6152489Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203404.xml (deflated 40%) 2022-03-17T21:37:01.6153277Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203410.xml (deflated 40%) 2022-03-17T21:37:01.6154101Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203416.xml (deflated 40%) 2022-03-17T21:37:01.6154770Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203423.xml (deflated 40%) 2022-03-17T21:37:01.6155589Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203431.xml (deflated 41%) 2022-03-17T21:37:01.6156299Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203432.xml (deflated 41%) 2022-03-17T21:37:01.6157049Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203433.xml (deflated 42%) 2022-03-17T21:37:01.6158001Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203434.xml (deflated 42%) 2022-03-17T21:37:01.6158688Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203435.xml (deflated 41%) 2022-03-17T21:37:01.6159635Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203436.xml (deflated 41%) 2022-03-17T21:37:01.6160404Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203438.xml (deflated 42%) 2022-03-17T21:37:01.6161220Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203439.xml (deflated 41%) 2022-03-17T21:37:01.6161921Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203440.xml (deflated 42%) 2022-03-17T21:37:01.6162721Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203441.xml (deflated 42%) 2022-03-17T21:37:01.6163437Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203442.xml (deflated 42%) 2022-03-17T21:37:01.6164210Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203443.xml (deflated 42%) 2022-03-17T21:37:01.6165112Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203444.xml (deflated 42%) 2022-03-17T21:37:01.6165796Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203446.xml (deflated 42%) 2022-03-17T21:37:01.6166562Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203447.xml (deflated 41%) 2022-03-17T21:37:01.6167366Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203448.xml (deflated 41%) 2022-03-17T21:37:01.6168074Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203449.xml (deflated 41%) 2022-03-17T21:37:01.6168883Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203450.xml (deflated 42%) 2022-03-17T21:37:01.6169607Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203451.xml (deflated 41%) 2022-03-17T21:37:01.6170407Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203453.xml (deflated 42%) 2022-03-17T21:37:01.6171114Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203454.xml (deflated 41%) 2022-03-17T21:37:01.6171913Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203455.xml (deflated 42%) 2022-03-17T21:37:01.6172668Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203456.xml (deflated 41%) 2022-03-17T21:37:01.6173425Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203457.xml (deflated 42%) 2022-03-17T21:37:01.6174148Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203458.xml (deflated 43%) 2022-03-17T21:37:01.6174984Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203459.xml (deflated 43%) 2022-03-17T21:37:01.6175713Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203501.xml (deflated 42%) 2022-03-17T21:37:01.6176625Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203502.xml (deflated 41%) 2022-03-17T21:37:01.6177307Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203508.xml (deflated 42%) 2022-03-17T21:37:01.6178139Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203509.xml (deflated 42%) 2022-03-17T21:37:01.6178825Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203511.xml (deflated 42%) 2022-03-17T21:37:01.6179645Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203512.xml (deflated 41%) 2022-03-17T21:37:01.6180620Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203518.xml (deflated 41%) 2022-03-17T21:37:01.6181341Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203524.xml (deflated 40%) 2022-03-17T21:37:01.6182108Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203530.xml (deflated 43%) 2022-03-17T21:37:01.6182907Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203531.xml (deflated 43%) 2022-03-17T21:37:01.6183626Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203533.xml (deflated 41%) 2022-03-17T21:37:01.6184459Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203539.xml (deflated 41%) 2022-03-17T21:37:01.6185259Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203546.xml (deflated 43%) 2022-03-17T21:37:01.6185991Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203547.xml (deflated 41%) 2022-03-17T21:37:01.6186843Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203554.xml (deflated 43%) 2022-03-17T21:37:01.6187503Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203555.xml (deflated 43%) 2022-03-17T21:37:01.6188305Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203559.xml (deflated 43%) 2022-03-17T21:37:01.6189023Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203600.xml (deflated 43%) 2022-03-17T21:37:01.6189778Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203602.xml (deflated 41%) 2022-03-17T21:37:01.6190532Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203610.xml (deflated 41%) 2022-03-17T21:37:01.6191318Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203618.xml (deflated 43%) 2022-03-17T21:37:01.6192139Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203619.xml (deflated 41%) 2022-03-17T21:37:01.6192826Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203627.xml (deflated 43%) 2022-03-17T21:37:01.6193682Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203628.xml (deflated 43%) 2022-03-17T21:37:01.6194344Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203632.xml (deflated 43%) 2022-03-17T21:37:01.6195165Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203633.xml (deflated 43%) 2022-03-17T21:37:01.6196040Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203635.xml (deflated 41%) 2022-03-17T21:37:01.6196708Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203641.xml (deflated 41%) 2022-03-17T21:37:01.6197489Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203648.xml (deflated 42%) 2022-03-17T21:37:01.6198240Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203649.xml (deflated 41%) 2022-03-17T21:37:01.6199005Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203656.xml (deflated 42%) 2022-03-17T21:37:01.6199835Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203657.xml (deflated 43%) 2022-03-17T21:37:01.6200499Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203701.xml (deflated 41%) 2022-03-17T21:37:01.6201336Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203708.xml (deflated 42%) 2022-03-17T21:37:01.6202030Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203712.xml (deflated 43%) 2022-03-17T21:37:01.6202913Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203713.xml (deflated 43%) 2022-03-17T21:37:01.6203597Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203714.xml (deflated 41%) 2022-03-17T21:37:01.6204370Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203722.xml (deflated 42%) 2022-03-17T21:37:01.6205105Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203723.xml (deflated 42%) 2022-03-17T21:37:01.6205884Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203727.xml (deflated 42%) 2022-03-17T21:37:01.6206718Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203728.xml (deflated 42%) 2022-03-17T21:37:01.6207397Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203732.xml (deflated 43%) 2022-03-17T21:37:01.6208240Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203734.xml (deflated 43%) 2022-03-17T21:37:01.6208915Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203735.xml (deflated 43%) 2022-03-17T21:37:01.6209680Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203736.xml (deflated 41%) 2022-03-17T21:37:01.6210450Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203737.xml (deflated 41%) 2022-03-17T21:37:01.6211201Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203738.xml (deflated 41%) 2022-03-17T21:37:01.6211964Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203743.xml (deflated 41%) 2022-03-17T21:37:01.6212707Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203748.xml (deflated 42%) 2022-03-17T21:37:01.6213458Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203752.xml (deflated 41%) 2022-03-17T21:37:01.6214308Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203758.xml (deflated 41%) 2022-03-17T21:37:01.6215092Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203802.xml (deflated 40%) 2022-03-17T21:37:01.6215864Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203808.xml (deflated 41%) 2022-03-17T21:37:01.6216588Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203812.xml (deflated 42%) 2022-03-17T21:37:01.6217369Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203814.xml (deflated 40%) 2022-03-17T21:37:01.6218103Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203820.xml (deflated 42%) 2022-03-17T21:37:01.6218879Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203821.xml (deflated 42%) 2022-03-17T21:37:01.6219716Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203822.xml (deflated 44%) 2022-03-17T21:37:01.6220568Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203823.xml (deflated 41%) 2022-03-17T21:37:01.6221316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203830.xml (deflated 41%) 2022-03-17T21:37:01.6222162Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203835.xml (deflated 41%) 2022-03-17T21:37:01.6222967Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203841.xml (deflated 40%) 2022-03-17T21:37:01.6223739Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203847.xml (deflated 41%) 2022-03-17T21:37:01.6224529Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203853.xml (deflated 40%) 2022-03-17T21:37:01.6225249Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203859.xml (deflated 41%) 2022-03-17T21:37:01.6226072Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203905.xml (deflated 40%) 2022-03-17T21:37:01.6226749Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203912.xml (deflated 40%) 2022-03-17T21:37:01.6227549Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203919.xml (deflated 40%) 2022-03-17T21:37:01.6228312Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203926.xml (deflated 41%) 2022-03-17T21:37:01.6229033Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203932.xml (deflated 41%) 2022-03-17T21:37:01.6229811Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203938.xml (deflated 43%) 2022-03-17T21:37:01.6230557Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203939.xml (deflated 41%) 2022-03-17T21:37:01.6231362Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203946.xml (deflated 41%) 2022-03-17T21:37:01.6232064Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317203953.xml (deflated 40%) 2022-03-17T21:37:01.6232840Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204002.xml (deflated 41%) 2022-03-17T21:37:01.6233746Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204009.xml (deflated 40%) 2022-03-17T21:37:01.6234440Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204016.xml (deflated 40%) 2022-03-17T21:37:01.6235191Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204023.xml (deflated 40%) 2022-03-17T21:37:01.6236040Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204031.xml (deflated 40%) 2022-03-17T21:37:01.6236722Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204039.xml (deflated 42%) 2022-03-17T21:37:01.6237512Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204040.xml (deflated 42%) 2022-03-17T21:37:01.6238223Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204041.xml (deflated 42%) 2022-03-17T21:37:01.6239004Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204042.xml (deflated 42%) 2022-03-17T21:37:01.6239727Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204043.xml (deflated 41%) 2022-03-17T21:37:01.6240541Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204044.xml (deflated 42%) 2022-03-17T21:37:01.6241399Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204045.xml (deflated 42%) 2022-03-17T21:37:01.6242079Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204047.xml (deflated 42%) 2022-03-17T21:37:01.6242918Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204048.xml (deflated 42%) 2022-03-17T21:37:01.6243603Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204049.xml (deflated 42%) 2022-03-17T21:37:01.6244423Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204050.xml (deflated 42%) 2022-03-17T21:37:01.6245122Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204051.xml (deflated 42%) 2022-03-17T21:37:01.6245961Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204052.xml (deflated 41%) 2022-03-17T21:37:01.6246629Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204059.xml (deflated 41%) 2022-03-17T21:37:01.6247465Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204106.xml (deflated 41%) 2022-03-17T21:37:01.6248145Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204113.xml (deflated 41%) 2022-03-17T21:37:01.6248930Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204114.xml (deflated 41%) 2022-03-17T21:37:01.6249662Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204121.xml (deflated 40%) 2022-03-17T21:37:01.6250488Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204127.xml (deflated 41%) 2022-03-17T21:37:01.6251174Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204143.xml (deflated 41%) 2022-03-17T21:37:01.6252018Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204150.xml (deflated 41%) 2022-03-17T21:37:01.6252738Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204158.xml (deflated 41%) 2022-03-17T21:37:01.6253511Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204205.xml (deflated 41%) 2022-03-17T21:37:01.6254270Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204212.xml (deflated 41%) 2022-03-17T21:37:01.6255017Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204219.xml (deflated 41%) 2022-03-17T21:37:01.6255768Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204225.xml (deflated 42%) 2022-03-17T21:37:01.6256537Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204232.xml (deflated 41%) 2022-03-17T21:37:01.6257364Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204239.xml (deflated 41%) 2022-03-17T21:37:01.6258050Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204245.xml (deflated 41%) 2022-03-17T21:37:01.6258777Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204251.xml (deflated 41%) 2022-03-17T21:37:01.6259602Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204257.xml (deflated 40%) 2022-03-17T21:37:01.6260585Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204303.xml (deflated 40%) 2022-03-17T21:37:01.6261379Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204310.xml (deflated 41%) 2022-03-17T21:37:01.6285496Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204317.xml (deflated 41%) 2022-03-17T21:37:01.6287271Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204324.xml (deflated 41%) 2022-03-17T21:37:01.6288185Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204528.xml (deflated 41%) 2022-03-17T21:37:01.6288875Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204534.xml (deflated 41%) 2022-03-17T21:37:01.6289669Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204541.xml (deflated 40%) 2022-03-17T21:37:01.6290445Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204545.xml (deflated 41%) 2022-03-17T21:37:01.6291508Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204549.xml (deflated 41%) 2022-03-17T21:37:01.6292894Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204555.xml (deflated 40%) 2022-03-17T21:37:01.6293626Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204601.xml (deflated 41%) 2022-03-17T21:37:01.6294424Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204606.xml (deflated 42%) 2022-03-17T21:37:01.6295208Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204607.xml (deflated 41%) 2022-03-17T21:37:01.6295981Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204608.xml (deflated 40%) 2022-03-17T21:37:01.6297054Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204616.xml (deflated 41%) 2022-03-17T21:37:01.6297975Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204617.xml (deflated 41%) 2022-03-17T21:37:01.6298702Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204618.xml (deflated 40%) 2022-03-17T21:37:01.6299462Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204625.xml (deflated 40%) 2022-03-17T21:37:01.6300519Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204632.xml (deflated 41%) 2022-03-17T21:37:01.6301293Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204637.xml (deflated 40%) 2022-03-17T21:37:01.6302090Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204643.xml (deflated 41%) 2022-03-17T21:37:01.6302841Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204648.xml (deflated 40%) 2022-03-17T21:37:01.6303629Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204652.xml (deflated 40%) 2022-03-17T21:37:01.6304400Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204656.xml (deflated 41%) 2022-03-17T21:37:01.6305236Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204702.xml (deflated 43%) 2022-03-17T21:37:01.6306021Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204703.xml (deflated 43%) 2022-03-17T21:37:01.6306795Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204705.xml (deflated 42%) 2022-03-17T21:37:01.6307552Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204706.xml (deflated 43%) 2022-03-17T21:37:01.6308316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204707.xml (deflated 41%) 2022-03-17T21:37:01.6309092Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204729.xml (deflated 41%) 2022-03-17T21:37:01.6309835Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204751.xml (deflated 42%) 2022-03-17T21:37:01.6310604Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204752.xml (deflated 42%) 2022-03-17T21:37:01.6311373Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204753.xml (deflated 42%) 2022-03-17T21:37:01.6312133Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204755.xml (deflated 42%) 2022-03-17T21:37:01.6312915Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204756.xml (deflated 41%) 2022-03-17T21:37:01.6313672Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204757.xml (deflated 40%) 2022-03-17T21:37:01.6314428Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204802.xml (deflated 41%) 2022-03-17T21:37:01.6315199Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204808.xml (deflated 41%) 2022-03-17T21:37:01.6315963Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204815.xml (deflated 41%) 2022-03-17T21:37:01.6316809Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204821.xml (deflated 40%) 2022-03-17T21:37:01.6317582Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204827.xml (deflated 42%) 2022-03-17T21:37:01.6318319Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204828.xml (deflated 42%) 2022-03-17T21:37:01.6319095Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204829.xml (deflated 42%) 2022-03-17T21:37:01.6319831Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204830.xml (deflated 41%) 2022-03-17T21:37:01.6320607Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204835.xml (deflated 41%) 2022-03-17T21:37:01.6321416Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204839.xml (deflated 42%) 2022-03-17T21:37:01.6322181Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204840.xml (deflated 43%) 2022-03-17T21:37:01.6322930Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204841.xml (deflated 41%) 2022-03-17T21:37:01.6323759Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204848.xml (deflated 41%) 2022-03-17T21:37:01.6324599Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204855.xml (deflated 40%) 2022-03-17T21:37:01.6325365Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204902.xml (deflated 40%) 2022-03-17T21:37:01.6326129Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204908.xml (deflated 41%) 2022-03-17T21:37:01.6326867Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204909.xml (deflated 41%) 2022-03-17T21:37:01.6327656Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204910.xml (deflated 42%) 2022-03-17T21:37:01.6328423Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204912.xml (deflated 41%) 2022-03-17T21:37:01.6329169Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204913.xml (deflated 41%) 2022-03-17T21:37:01.6329901Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204914.xml (deflated 41%) 2022-03-17T21:37:01.6330695Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204915.xml (deflated 41%) 2022-03-17T21:37:01.6331445Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204916.xml (deflated 41%) 2022-03-17T21:37:01.6332212Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204917.xml (deflated 42%) 2022-03-17T21:37:01.6332978Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204918.xml (deflated 42%) 2022-03-17T21:37:01.6333729Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204920.xml (deflated 40%) 2022-03-17T21:37:01.6334487Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204926.xml (deflated 42%) 2022-03-17T21:37:01.6335241Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204927.xml (deflated 42%) 2022-03-17T21:37:01.6336062Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204928.xml (deflated 41%) 2022-03-17T21:37:01.6336831Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204934.xml (deflated 41%) 2022-03-17T21:37:01.6337603Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204941.xml (deflated 42%) 2022-03-17T21:37:01.6338344Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204942.xml (deflated 42%) 2022-03-17T21:37:01.6339111Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204943.xml (deflated 41%) 2022-03-17T21:37:01.6339867Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204944.xml (deflated 42%) 2022-03-17T21:37:01.6340798Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204945.xml (deflated 41%) 2022-03-17T21:37:01.6341522Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317204953.xml (deflated 41%) 2022-03-17T21:37:01.6342230Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205001.xml (deflated 42%) 2022-03-17T21:37:01.6343052Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205002.xml (deflated 42%) 2022-03-17T21:37:01.6343819Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205003.xml (deflated 42%) 2022-03-17T21:37:01.6344569Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205005.xml (deflated 44%) 2022-03-17T21:37:01.6345337Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205006.xml (deflated 43%) 2022-03-17T21:37:01.6346100Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205007.xml (deflated 43%) 2022-03-17T21:37:01.6346848Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205008.xml (deflated 43%) 2022-03-17T21:37:01.6347618Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205009.xml (deflated 43%) 2022-03-17T21:37:01.6348385Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205010.xml (deflated 41%) 2022-03-17T21:37:01.6349135Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205016.xml (deflated 41%) 2022-03-17T21:37:01.6349904Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205022.xml (deflated 41%) 2022-03-17T21:37:01.6350666Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205028.xml (deflated 44%) 2022-03-17T21:37:01.6351412Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205030.xml (deflated 44%) 2022-03-17T21:37:01.6352181Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205031.xml (deflated 43%) 2022-03-17T21:37:01.6353235Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205032.xml (deflated 43%) 2022-03-17T21:37:01.6354015Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205033.xml (deflated 43%) 2022-03-17T21:37:01.6354770Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205034.xml (deflated 43%) 2022-03-17T21:37:01.6355606Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205035.xml (deflated 41%) 2022-03-17T21:37:01.6356368Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205042.xml (deflated 42%) 2022-03-17T21:37:01.6357128Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205044.xml (deflated 40%) 2022-03-17T21:37:01.6357880Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205050.xml (deflated 41%) 2022-03-17T21:37:01.6358643Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205057.xml (deflated 40%) 2022-03-17T21:37:01.6359408Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205108.xml (deflated 40%) 2022-03-17T21:37:01.6360152Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205121.xml (deflated 42%) 2022-03-17T21:37:01.6360913Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205127.xml (deflated 42%) 2022-03-17T21:37:01.6361677Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205128.xml (deflated 42%) 2022-03-17T21:37:01.6362484Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205133.xml (deflated 41%) 2022-03-17T21:37:01.6363262Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205141.xml (deflated 41%) 2022-03-17T21:37:01.6364028Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205145.xml (deflated 41%) 2022-03-17T21:37:01.6364780Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205149.xml (deflated 41%) 2022-03-17T21:37:01.6365547Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205156.xml (deflated 40%) 2022-03-17T21:37:01.6366301Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205202.xml (deflated 41%) 2022-03-17T21:37:01.6367070Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205208.xml (deflated 39%) 2022-03-17T21:37:01.6367836Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205213.xml (deflated 40%) 2022-03-17T21:37:01.6368584Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205220.xml (deflated 40%) 2022-03-17T21:37:01.6369354Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205226.xml (deflated 40%) 2022-03-17T21:37:01.6370118Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205231.xml (deflated 41%) 2022-03-17T21:37:01.6370871Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205239.xml (deflated 41%) 2022-03-17T21:37:01.6371638Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205244.xml (deflated 41%) 2022-03-17T21:37:01.6372396Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205250.xml (deflated 42%) 2022-03-17T21:37:01.6373145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205256.xml (deflated 42%) 2022-03-17T21:37:01.6373991Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205257.xml (deflated 46%) 2022-03-17T21:37:01.6374749Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205301.xml (deflated 47%) 2022-03-17T21:37:01.6375497Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205305.xml (deflated 48%) 2022-03-17T21:37:01.6376268Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205310.xml (deflated 46%) 2022-03-17T21:37:01.6377013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205314.xml (deflated 41%) 2022-03-17T21:37:01.6377779Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205318.xml (deflated 40%) 2022-03-17T21:37:01.6378546Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205322.xml (deflated 41%) 2022-03-17T21:37:01.6379294Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205326.xml (deflated 42%) 2022-03-17T21:37:01.6380057Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205331.xml (deflated 40%) 2022-03-17T21:37:01.6381033Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205335.xml (deflated 41%) 2022-03-17T21:37:01.6381856Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205339.xml (deflated 40%) 2022-03-17T21:37:01.6382638Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205344.xml (deflated 42%) 2022-03-17T21:37:01.6383395Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205345.xml (deflated 41%) 2022-03-17T21:37:01.6384149Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205346.xml (deflated 40%) 2022-03-17T21:37:01.6384913Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205350.xml (deflated 42%) 2022-03-17T21:37:01.6385672Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205354.xml (deflated 43%) 2022-03-17T21:37:01.6386426Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205355.xml (deflated 42%) 2022-03-17T21:37:01.6386864Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205356.xml (deflated 40%) 2022-03-17T21:37:01.6387303Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205401.xml (deflated 40%) 2022-03-17T21:37:01.6387737Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205405.xml (deflated 41%) 2022-03-17T21:37:01.6388170Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205409.xml (deflated 41%) 2022-03-17T21:37:01.6388584Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205413.xml (deflated 41%) 2022-03-17T21:37:01.6389020Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205418.xml (deflated 40%) 2022-03-17T21:37:01.6389455Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205422.xml (deflated 42%) 2022-03-17T21:37:01.6389892Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205426.xml (deflated 42%) 2022-03-17T21:37:01.6390414Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205430.xml (deflated 42%) 2022-03-17T21:37:01.6390847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205435.xml (deflated 42%) 2022-03-17T21:37:01.6391280Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205439.xml (deflated 41%) 2022-03-17T21:37:01.6391702Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205443.xml (deflated 41%) 2022-03-17T21:37:01.6392139Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205447.xml (deflated 41%) 2022-03-17T21:37:01.6392571Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205452.xml (deflated 41%) 2022-03-17T21:37:01.6393008Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205456.xml (deflated 41%) 2022-03-17T21:37:01.6393439Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205500.xml (deflated 41%) 2022-03-17T21:37:01.6393872Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205504.xml (deflated 40%) 2022-03-17T21:37:01.6394355Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205509.xml (deflated 40%) 2022-03-17T21:37:01.6394778Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205513.xml (deflated 41%) 2022-03-17T21:37:01.6395204Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205517.xml (deflated 41%) 2022-03-17T21:37:01.6395639Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205522.xml (deflated 42%) 2022-03-17T21:37:01.6396073Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205526.xml (deflated 42%) 2022-03-17T21:37:01.6396503Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205530.xml (deflated 43%) 2022-03-17T21:37:01.6396936Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205534.xml (deflated 42%) 2022-03-17T21:37:01.6397352Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205538.xml (deflated 41%) 2022-03-17T21:37:01.6397784Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205542.xml (deflated 41%) 2022-03-17T21:37:01.6398220Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205547.xml (deflated 41%) 2022-03-17T21:37:01.6398651Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205553.xml (deflated 40%) 2022-03-17T21:37:01.6399084Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205559.xml (deflated 41%) 2022-03-17T21:37:01.6399513Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205603.xml (deflated 41%) 2022-03-17T21:37:01.6399949Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205609.xml (deflated 41%) 2022-03-17T21:37:01.6400366Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205613.xml (deflated 40%) 2022-03-17T21:37:01.6400798Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205617.xml (deflated 41%) 2022-03-17T21:37:01.6401302Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205621.xml (deflated 41%) 2022-03-17T21:37:01.6401733Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205629.xml (deflated 40%) 2022-03-17T21:37:01.6402162Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205638.xml (deflated 40%) 2022-03-17T21:37:01.6402596Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205646.xml (deflated 43%) 2022-03-17T21:37:01.6403011Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205647.xml (deflated 42%) 2022-03-17T21:37:01.6403443Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205648.xml (deflated 42%) 2022-03-17T21:37:01.6403877Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205649.xml (deflated 42%) 2022-03-17T21:37:01.6404310Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205650.xml (deflated 43%) 2022-03-17T21:37:01.6404736Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205651.xml (deflated 42%) 2022-03-17T21:37:01.6405210Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205653.xml (deflated 43%) 2022-03-17T21:37:01.6405650Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205654.xml (deflated 42%) 2022-03-17T21:37:01.6406064Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205655.xml (deflated 43%) 2022-03-17T21:37:01.6406499Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205656.xml (deflated 42%) 2022-03-17T21:37:01.6406929Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205657.xml (deflated 43%) 2022-03-17T21:37:01.6407356Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205658.xml (deflated 42%) 2022-03-17T21:37:01.6407791Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205700.xml (deflated 43%) 2022-03-17T21:37:01.6408222Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205701.xml (deflated 43%) 2022-03-17T21:37:01.6408653Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205702.xml (deflated 43%) 2022-03-17T21:37:01.6409070Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205703.xml (deflated 43%) 2022-03-17T21:37:01.6409498Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205704.xml (deflated 43%) 2022-03-17T21:37:01.6409928Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205705.xml (deflated 42%) 2022-03-17T21:37:01.6410361Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205706.xml (deflated 43%) 2022-03-17T21:37:01.6410793Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205707.xml (deflated 43%) 2022-03-17T21:37:01.6411221Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205709.xml (deflated 42%) 2022-03-17T21:37:01.6411697Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205710.xml (deflated 43%) 2022-03-17T21:37:01.6412128Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205711.xml (deflated 42%) 2022-03-17T21:37:01.6412555Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205712.xml (deflated 43%) 2022-03-17T21:37:01.6412987Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205713.xml (deflated 41%) 2022-03-17T21:37:01.6413419Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205720.xml (deflated 42%) 2022-03-17T21:37:01.6413847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205724.xml (deflated 43%) 2022-03-17T21:37:01.6414274Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205725.xml (deflated 41%) 2022-03-17T21:37:01.6414693Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205730.xml (deflated 41%) 2022-03-17T21:37:01.6415124Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205737.xml (deflated 40%) 2022-03-17T21:37:01.6415558Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205742.xml (deflated 42%) 2022-03-17T21:37:01.6416031Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205746.xml (deflated 42%) 2022-03-17T21:37:01.6416471Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205750.xml (deflated 42%) 2022-03-17T21:37:01.6416899Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205754.xml (deflated 40%) 2022-03-17T21:37:01.6417332Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205759.xml (deflated 42%) 2022-03-17T21:37:01.6417746Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205801.xml (deflated 42%) 2022-03-17T21:37:01.6418180Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205805.xml (deflated 41%) 2022-03-17T21:37:01.6418616Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205809.xml (deflated 40%) 2022-03-17T21:37:01.6419048Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205813.xml (deflated 42%) 2022-03-17T21:37:01.6419477Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205815.xml (deflated 41%) 2022-03-17T21:37:01.6419910Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205816.xml (deflated 41%) 2022-03-17T21:37:01.6420505Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205817.xml (deflated 41%) 2022-03-17T21:37:01.6420950Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205818.xml (deflated 41%) 2022-03-17T21:37:01.6421382Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205819.xml (deflated 41%) 2022-03-17T21:37:01.6421810Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205820.xml (deflated 42%) 2022-03-17T21:37:01.6422237Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205821.xml (deflated 41%) 2022-03-17T21:37:01.6422751Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205826.xml (deflated 40%) 2022-03-17T21:37:01.6423222Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205832.xml (deflated 40%) 2022-03-17T21:37:01.6423637Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205836.xml (deflated 42%) 2022-03-17T21:37:01.6424067Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205840.xml (deflated 40%) 2022-03-17T21:37:01.6424499Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205846.xml (deflated 40%) 2022-03-17T21:37:01.6424931Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205850.xml (deflated 41%) 2022-03-17T21:37:01.6425365Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205856.xml (deflated 41%) 2022-03-17T21:37:01.6425796Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205902.xml (deflated 40%) 2022-03-17T21:37:01.6426211Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205908.xml (deflated 40%) 2022-03-17T21:37:01.6426717Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205914.xml (deflated 40%) 2022-03-17T21:37:01.6427169Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205919.xml (deflated 41%) 2022-03-17T21:37:01.6427599Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205925.xml (deflated 41%) 2022-03-17T21:37:01.6428027Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205931.xml (deflated 40%) 2022-03-17T21:37:01.6428450Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205937.xml (deflated 40%) 2022-03-17T21:37:01.6428879Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205943.xml (deflated 40%) 2022-03-17T21:37:01.6429295Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205949.xml (deflated 40%) 2022-03-17T21:37:01.6429727Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205955.xml (deflated 41%) 2022-03-17T21:37:01.6430156Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317205959.xml (deflated 41%) 2022-03-17T21:37:01.6430588Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210005.xml (deflated 41%) 2022-03-17T21:37:01.6431022Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210011.xml (deflated 41%) 2022-03-17T21:37:01.6431453Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210023.xml (deflated 41%) 2022-03-17T21:37:01.6431883Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210029.xml (deflated 41%) 2022-03-17T21:37:01.6432300Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210035.xml (deflated 40%) 2022-03-17T21:37:01.6432728Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210041.xml (deflated 40%) 2022-03-17T21:37:01.6433160Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210048.xml (deflated 40%) 2022-03-17T21:37:01.6433652Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210054.xml (deflated 41%) 2022-03-17T21:37:01.6434078Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210101.xml (deflated 41%) 2022-03-17T21:37:01.6434507Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210107.xml (deflated 41%) 2022-03-17T21:37:01.6434925Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210113.xml (deflated 41%) 2022-03-17T21:37:01.6435360Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210119.xml (deflated 41%) 2022-03-17T21:37:01.6435790Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210125.xml (deflated 41%) 2022-03-17T21:37:01.6436224Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210132.xml (deflated 41%) 2022-03-17T21:37:01.6436656Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210138.xml (deflated 41%) 2022-03-17T21:37:01.6437085Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210144.xml (deflated 41%) 2022-03-17T21:37:01.6437563Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210150.xml (deflated 41%) 2022-03-17T21:37:01.6437988Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210156.xml (deflated 41%) 2022-03-17T21:37:01.6438417Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210203.xml (deflated 41%) 2022-03-17T21:37:01.6438851Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210209.xml (deflated 41%) 2022-03-17T21:37:01.6439283Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210215.xml (deflated 41%) 2022-03-17T21:37:01.6439714Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210221.xml (deflated 41%) 2022-03-17T21:37:01.6440145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210227.xml (deflated 41%) 2022-03-17T21:37:01.6440576Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210231.xml (deflated 40%) 2022-03-17T21:37:01.6440990Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210238.xml (deflated 41%) 2022-03-17T21:37:01.6441426Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210243.xml (deflated 41%) 2022-03-17T21:37:01.6441856Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210259.xml (deflated 41%) 2022-03-17T21:37:01.6442286Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210305.xml (deflated 41%) 2022-03-17T21:37:01.6442718Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210312.xml (deflated 41%) 2022-03-17T21:37:01.6443146Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210318.xml (deflated 41%) 2022-03-17T21:37:01.6443561Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210324.xml (deflated 41%) 2022-03-17T21:37:01.6444055Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210330.xml (deflated 42%) 2022-03-17T21:37:01.6444484Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210336.xml (deflated 42%) 2022-03-17T21:37:01.6444911Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210343.xml (deflated 41%) 2022-03-17T21:37:01.6445338Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210348.xml (deflated 41%) 2022-03-17T21:37:01.6445773Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210354.xml (deflated 40%) 2022-03-17T21:37:01.6446201Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210400.xml (deflated 41%) 2022-03-17T21:37:01.6446615Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210406.xml (deflated 40%) 2022-03-17T21:37:01.6447049Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210413.xml (deflated 40%) 2022-03-17T21:37:01.6447478Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210419.xml (deflated 40%) 2022-03-17T21:37:01.6447911Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210424.xml (deflated 41%) 2022-03-17T21:37:01.6448395Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210430.xml (deflated 41%) 2022-03-17T21:37:01.6448837Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210636.xml (deflated 41%) 2022-03-17T21:37:01.6449249Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210642.xml (deflated 40%) 2022-03-17T21:37:01.6449684Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210648.xml (deflated 40%) 2022-03-17T21:37:01.6450110Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210652.xml (deflated 41%) 2022-03-17T21:37:01.6450534Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210656.xml (deflated 41%) 2022-03-17T21:37:01.6450968Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210702.xml (deflated 41%) 2022-03-17T21:37:01.6451401Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210708.xml (deflated 41%) 2022-03-17T21:37:01.6452000Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210713.xml (deflated 41%) 2022-03-17T21:37:01.6452420Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210717.xml (deflated 41%) 2022-03-17T21:37:01.6452853Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210721.xml (deflated 42%) 2022-03-17T21:37:01.6453284Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210722.xml (deflated 41%) 2022-03-17T21:37:01.6453718Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210727.xml (deflated 43%) 2022-03-17T21:37:01.6454148Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210731.xml (deflated 41%) 2022-03-17T21:37:01.6454579Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210735.xml (deflated 40%) 2022-03-17T21:37:01.6455074Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210739.xml (deflated 41%) 2022-03-17T21:37:01.6455488Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210744.xml (deflated 40%) 2022-03-17T21:37:01.6455917Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210748.xml (deflated 41%) 2022-03-17T21:37:01.6456351Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210752.xml (deflated 41%) 2022-03-17T21:37:01.6456783Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210757.xml (deflated 41%) 2022-03-17T21:37:01.6457212Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210801.xml (deflated 41%) 2022-03-17T21:37:01.6457644Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210807.xml (deflated 41%) 2022-03-17T21:37:01.6458074Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210811.xml (deflated 41%) 2022-03-17T21:37:01.6458484Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210815.xml (deflated 41%) 2022-03-17T21:37:01.6458970Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210820.xml (deflated 41%) 2022-03-17T21:37:01.6459420Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210824.xml (deflated 41%) 2022-03-17T21:37:01.6459851Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210846.xml (deflated 41%) 2022-03-17T21:37:01.6460480Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210909.xml (deflated 43%) 2022-03-17T21:37:01.6460909Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210913.xml (deflated 41%) 2022-03-17T21:37:01.6461320Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210919.xml (deflated 40%) 2022-03-17T21:37:01.6461754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210923.xml (deflated 40%) 2022-03-17T21:37:01.6462189Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210928.xml (deflated 42%) 2022-03-17T21:37:01.6462619Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210932.xml (deflated 42%) 2022-03-17T21:37:01.6463050Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210933.xml (deflated 42%) 2022-03-17T21:37:01.6463480Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210934.xml (deflated 42%) 2022-03-17T21:37:01.6463906Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210935.xml (deflated 42%) 2022-03-17T21:37:01.6464321Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210936.xml (deflated 43%) 2022-03-17T21:37:01.6464756Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210937.xml (deflated 42%) 2022-03-17T21:37:01.6465190Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210939.xml (deflated 42%) 2022-03-17T21:37:01.6465620Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210940.xml (deflated 42%) 2022-03-17T21:37:01.6466145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210941.xml (deflated 40%) 2022-03-17T21:37:01.6466576Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210945.xml (deflated 41%) 2022-03-17T21:37:01.6467008Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210949.xml (deflated 42%) 2022-03-17T21:37:01.6467424Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210951.xml (deflated 43%) 2022-03-17T21:37:01.6467855Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210952.xml (deflated 41%) 2022-03-17T21:37:01.6468286Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317210958.xml (deflated 41%) 2022-03-17T21:37:01.6468723Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211004.xml (deflated 41%) 2022-03-17T21:37:01.6469151Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211010.xml (deflated 40%) 2022-03-17T21:37:01.6469584Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211016.xml (deflated 41%) 2022-03-17T21:37:01.6470088Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211021.xml (deflated 41%) 2022-03-17T21:37:01.6470540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211025.xml (deflated 41%) 2022-03-17T21:37:01.6470972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211029.xml (deflated 41%) 2022-03-17T21:37:01.6471405Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211033.xml (deflated 42%) 2022-03-17T21:37:01.6471833Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211038.xml (deflated 42%) 2022-03-17T21:37:01.6472263Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211042.xml (deflated 42%) 2022-03-17T21:37:01.6472693Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211046.xml (deflated 42%) 2022-03-17T21:37:01.6473109Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211050.xml (deflated 41%) 2022-03-17T21:37:01.6473538Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211055.xml (deflated 41%) 2022-03-17T21:37:01.6473973Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211059.xml (deflated 43%) 2022-03-17T21:37:01.6474403Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211100.xml (deflated 41%) 2022-03-17T21:37:01.6474833Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211104.xml (deflated 41%) 2022-03-17T21:37:01.6475268Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211108.xml (deflated 43%) 2022-03-17T21:37:01.6475753Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211110.xml (deflated 42%) 2022-03-17T21:37:01.6476187Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211111.xml (deflated 41%) 2022-03-17T21:37:01.6476679Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211115.xml (deflated 41%) 2022-03-17T21:37:01.6477111Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211119.xml (deflated 41%) 2022-03-17T21:37:01.6477542Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211123.xml (deflated 41%) 2022-03-17T21:37:01.6477974Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211128.xml (deflated 41%) 2022-03-17T21:37:01.6478408Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211129.xml (deflated 41%) 2022-03-17T21:37:01.6478822Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211130.xml (deflated 41%) 2022-03-17T21:37:01.6479252Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211134.xml (deflated 42%) 2022-03-17T21:37:01.6479683Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211138.xml (deflated 41%) 2022-03-17T21:37:01.6480110Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211143.xml (deflated 41%) 2022-03-17T21:37:01.6480539Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211147.xml (deflated 41%) 2022-03-17T21:37:01.6481012Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211151.xml (deflated 41%) 2022-03-17T21:37:01.6481452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211155.xml (deflated 41%) 2022-03-17T21:37:01.6481868Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211200.xml (deflated 41%) 2022-03-17T21:37:01.6482302Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211204.xml (deflated 42%) 2022-03-17T21:37:01.6482735Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211205.xml (deflated 42%) 2022-03-17T21:37:01.6483165Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211206.xml (deflated 42%) 2022-03-17T21:37:01.6483598Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211207.xml (deflated 41%) 2022-03-17T21:37:01.6484028Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211212.xml (deflated 41%) 2022-03-17T21:37:01.6484441Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211216.xml (deflated 41%) 2022-03-17T21:37:01.6484878Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211220.xml (deflated 41%) 2022-03-17T21:37:01.6485311Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211224.xml (deflated 41%) 2022-03-17T21:37:01.6485742Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211228.xml (deflated 41%) 2022-03-17T21:37:01.6486176Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211234.xml (deflated 41%) 2022-03-17T21:37:01.6486606Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211240.xml (deflated 41%) 2022-03-17T21:37:01.6487035Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211244.xml (deflated 40%) 2022-03-17T21:37:01.6487510Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211250.xml (deflated 41%) 2022-03-17T21:37:01.6487938Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211256.xml (deflated 40%) 2022-03-17T21:37:01.6488367Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211307.xml (deflated 40%) 2022-03-17T21:37:01.6488803Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211319.xml (deflated 42%) 2022-03-17T21:37:01.6489232Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211325.xml (deflated 42%) 2022-03-17T21:37:01.6489663Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211326.xml (deflated 42%) 2022-03-17T21:37:01.6490098Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211331.xml (deflated 42%) 2022-03-17T21:37:01.6490514Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211337.xml (deflated 41%) 2022-03-17T21:37:01.6490945Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211342.xml (deflated 41%) 2022-03-17T21:37:01.6491424Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211346.xml (deflated 40%) 2022-03-17T21:37:01.6491862Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211353.xml (deflated 40%) 2022-03-17T21:37:01.6492289Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211358.xml (deflated 40%) 2022-03-17T21:37:01.6492723Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211404.xml (deflated 39%) 2022-03-17T21:37:01.6493139Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211410.xml (deflated 40%) 2022-03-17T21:37:01.6493572Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211416.xml (deflated 40%) 2022-03-17T21:37:01.6494000Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211422.xml (deflated 40%) 2022-03-17T21:37:01.6494432Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211428.xml (deflated 41%) 2022-03-17T21:37:01.6494863Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211436.xml (deflated 41%) 2022-03-17T21:37:01.6495293Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211440.xml (deflated 41%) 2022-03-17T21:37:01.6495726Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211446.xml (deflated 42%) 2022-03-17T21:37:01.6496141Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211452.xml (deflated 42%) 2022-03-17T21:37:01.6496570Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211453.xml (deflated 46%) 2022-03-17T21:37:01.6497002Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211457.xml (deflated 47%) 2022-03-17T21:37:01.6497435Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211501.xml (deflated 48%) 2022-03-17T21:37:01.6497868Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211506.xml (deflated 46%) 2022-03-17T21:37:01.6498358Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211510.xml (deflated 41%) 2022-03-17T21:37:01.6498788Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211514.xml (deflated 41%) 2022-03-17T21:37:01.6499199Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211518.xml (deflated 41%) 2022-03-17T21:37:01.6499634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211522.xml (deflated 42%) 2022-03-17T21:37:01.6500065Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211527.xml (deflated 41%) 2022-03-17T21:37:01.6500677Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211531.xml (deflated 41%) 2022-03-17T21:37:01.6501121Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211535.xml (deflated 40%) 2022-03-17T21:37:01.6501548Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211539.xml (deflated 42%) 2022-03-17T21:37:01.6501964Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211540.xml (deflated 41%) 2022-03-17T21:37:01.6502477Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211542.xml (deflated 40%) 2022-03-17T21:37:01.6502926Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211546.xml (deflated 42%) 2022-03-17T21:37:01.6503354Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211550.xml (deflated 43%) 2022-03-17T21:37:01.6503789Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211551.xml (deflated 42%) 2022-03-17T21:37:01.6504217Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211552.xml (deflated 40%) 2022-03-17T21:37:01.6504646Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211556.xml (deflated 40%) 2022-03-17T21:37:01.6505062Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211601.xml (deflated 41%) 2022-03-17T21:37:01.6505492Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211605.xml (deflated 41%) 2022-03-17T21:37:01.6505921Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211609.xml (deflated 41%) 2022-03-17T21:37:01.6506359Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211614.xml (deflated 41%) 2022-03-17T21:37:01.6506788Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211618.xml (deflated 42%) 2022-03-17T21:37:01.6507218Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211622.xml (deflated 42%) 2022-03-17T21:37:01.6507634Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211626.xml (deflated 42%) 2022-03-17T21:37:01.6508061Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211631.xml (deflated 42%) 2022-03-17T21:37:01.6508481Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211635.xml (deflated 41%) 2022-03-17T21:37:01.6508984Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211639.xml (deflated 41%) 2022-03-17T21:37:01.6509416Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211643.xml (deflated 41%) 2022-03-17T21:37:01.6509845Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211647.xml (deflated 41%) 2022-03-17T21:37:01.6510274Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211652.xml (deflated 41%) 2022-03-17T21:37:01.6510690Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211656.xml (deflated 41%) 2022-03-17T21:37:01.6511123Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211700.xml (deflated 41%) 2022-03-17T21:37:01.6511553Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211704.xml (deflated 41%) 2022-03-17T21:37:01.6511983Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211709.xml (deflated 41%) 2022-03-17T21:37:01.6512415Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211713.xml (deflated 41%) 2022-03-17T21:37:01.6512844Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211717.xml (deflated 42%) 2022-03-17T21:37:01.6513308Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211721.xml (deflated 43%) 2022-03-17T21:37:01.6513731Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211725.xml (deflated 42%) 2022-03-17T21:37:01.6514158Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211730.xml (deflated 43%) 2022-03-17T21:37:01.6514594Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211734.xml (deflated 41%) 2022-03-17T21:37:01.6515024Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211738.xml (deflated 41%) 2022-03-17T21:37:01.6515452Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211742.xml (deflated 40%) 2022-03-17T21:37:01.6515891Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211748.xml (deflated 40%) 2022-03-17T21:37:01.6516305Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211754.xml (deflated 41%) 2022-03-17T21:37:01.6516735Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211759.xml (deflated 41%) 2022-03-17T21:37:01.6517171Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211804.xml (deflated 41%) 2022-03-17T21:37:01.6517602Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211808.xml (deflated 41%) 2022-03-17T21:37:01.6518029Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211813.xml (deflated 41%) 2022-03-17T21:37:01.6518465Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211817.xml (deflated 41%) 2022-03-17T21:37:01.6518899Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211825.xml (deflated 41%) 2022-03-17T21:37:01.6519310Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211833.xml (deflated 40%) 2022-03-17T21:37:01.6519795Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211841.xml (deflated 43%) 2022-03-17T21:37:01.6520228Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211842.xml (deflated 42%) 2022-03-17T21:37:01.6520659Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211843.xml (deflated 42%) 2022-03-17T21:37:01.6521091Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211845.xml (deflated 42%) 2022-03-17T21:37:01.6521522Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211846.xml (deflated 43%) 2022-03-17T21:37:01.6521951Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211847.xml (deflated 42%) 2022-03-17T21:37:01.6522369Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211848.xml (deflated 43%) 2022-03-17T21:37:01.6522800Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211849.xml (deflated 42%) 2022-03-17T21:37:01.6523273Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211850.xml (deflated 43%) 2022-03-17T21:37:01.6523754Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211852.xml (deflated 42%) 2022-03-17T21:37:01.6524190Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211853.xml (deflated 43%) 2022-03-17T21:37:01.6524622Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211854.xml (deflated 43%) 2022-03-17T21:37:01.6525039Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211855.xml (deflated 42%) 2022-03-17T21:37:01.6525469Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211856.xml (deflated 43%) 2022-03-17T21:37:01.6525897Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211857.xml (deflated 42%) 2022-03-17T21:37:01.6526329Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211858.xml (deflated 43%) 2022-03-17T21:37:01.6526762Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211900.xml (deflated 43%) 2022-03-17T21:37:01.6527193Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211901.xml (deflated 42%) 2022-03-17T21:37:01.6527620Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211902.xml (deflated 43%) 2022-03-17T21:37:01.6528040Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211903.xml (deflated 43%) 2022-03-17T21:37:01.6528473Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211904.xml (deflated 42%) 2022-03-17T21:37:01.6528908Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211905.xml (deflated 42%) 2022-03-17T21:37:01.6529345Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211906.xml (deflated 42%) 2022-03-17T21:37:01.6529777Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211908.xml (deflated 43%) 2022-03-17T21:37:01.6530198Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211909.xml (deflated 41%) 2022-03-17T21:37:01.6530673Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211915.xml (deflated 42%) 2022-03-17T21:37:01.6531106Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211920.xml (deflated 43%) 2022-03-17T21:37:01.6531537Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211921.xml (deflated 41%) 2022-03-17T21:37:01.6531971Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211926.xml (deflated 41%) 2022-03-17T21:37:01.6532400Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211932.xml (deflated 40%) 2022-03-17T21:37:01.6532828Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211937.xml (deflated 42%) 2022-03-17T21:37:01.6533270Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211942.xml (deflated 42%) 2022-03-17T21:37:01.6533684Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211946.xml (deflated 42%) 2022-03-17T21:37:01.6534118Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211950.xml (deflated 40%) 2022-03-17T21:37:01.6534589Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317211956.xml (deflated 41%) 2022-03-17T21:37:01.6535027Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212001.xml (deflated 43%) 2022-03-17T21:37:01.6535456Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212005.xml (deflated 41%) 2022-03-17T21:37:01.6535891Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212009.xml (deflated 41%) 2022-03-17T21:37:01.6536321Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212014.xml (deflated 42%) 2022-03-17T21:37:01.6536738Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212015.xml (deflated 42%) 2022-03-17T21:37:01.6537170Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212016.xml (deflated 41%) 2022-03-17T21:37:01.6537601Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212017.xml (deflated 42%) 2022-03-17T21:37:01.6538033Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212018.xml (deflated 42%) 2022-03-17T21:37:01.6538466Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212020.xml (deflated 42%) 2022-03-17T21:37:01.6538896Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212021.xml (deflated 42%) 2022-03-17T21:37:01.6539309Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212022.xml (deflated 41%) 2022-03-17T21:37:01.6539742Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212026.xml (deflated 41%) 2022-03-17T21:37:01.6540172Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212032.xml (deflated 40%) 2022-03-17T21:37:01.6540792Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212037.xml (deflated 42%) 2022-03-17T21:37:01.6541310Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212041.xml (deflated 40%) 2022-03-17T21:37:01.6541739Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212046.xml (deflated 40%) 2022-03-17T21:37:01.6542169Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212051.xml (deflated 41%) 2022-03-17T21:37:01.6542583Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212056.xml (deflated 41%) 2022-03-17T21:37:01.6543013Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212102.xml (deflated 41%) 2022-03-17T21:37:01.6543441Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212108.xml (deflated 41%) 2022-03-17T21:37:01.6543868Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212114.xml (deflated 40%) 2022-03-17T21:37:01.6544299Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212120.xml (deflated 41%) 2022-03-17T21:37:01.6544731Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212126.xml (deflated 41%) 2022-03-17T21:37:01.6545162Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212132.xml (deflated 40%) 2022-03-17T21:37:01.6545633Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212137.xml (deflated 40%) 2022-03-17T21:37:01.6546080Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212143.xml (deflated 41%) 2022-03-17T21:37:01.6546511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212149.xml (deflated 41%) 2022-03-17T21:37:01.6546946Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212155.xml (deflated 41%) 2022-03-17T21:37:01.6547374Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212159.xml (deflated 41%) 2022-03-17T21:37:01.6547802Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212205.xml (deflated 41%) 2022-03-17T21:37:01.6548216Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212211.xml (deflated 41%) 2022-03-17T21:37:01.6548647Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212223.xml (deflated 41%) 2022-03-17T21:37:01.6549075Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212229.xml (deflated 41%) 2022-03-17T21:37:01.6549511Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212235.xml (deflated 41%) 2022-03-17T21:37:01.6549939Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212242.xml (deflated 40%) 2022-03-17T21:37:01.6550367Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212248.xml (deflated 40%) 2022-03-17T21:37:01.6550793Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212255.xml (deflated 41%) 2022-03-17T21:37:01.6551205Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212301.xml (deflated 41%) 2022-03-17T21:37:01.6551633Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212307.xml (deflated 41%) 2022-03-17T21:37:01.6552122Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212314.xml (deflated 41%) 2022-03-17T21:37:01.6552549Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212320.xml (deflated 41%) 2022-03-17T21:37:01.6552977Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212326.xml (deflated 41%) 2022-03-17T21:37:01.6553405Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212332.xml (deflated 41%) 2022-03-17T21:37:01.6553821Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212338.xml (deflated 41%) 2022-03-17T21:37:01.6554251Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212345.xml (deflated 41%) 2022-03-17T21:37:01.6554687Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212351.xml (deflated 41%) 2022-03-17T21:37:01.6555120Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212357.xml (deflated 41%) 2022-03-17T21:37:01.6555548Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212403.xml (deflated 41%) 2022-03-17T21:37:01.6556020Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212409.xml (deflated 40%) 2022-03-17T21:37:01.6556459Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212415.xml (deflated 41%) 2022-03-17T21:37:01.6556871Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212422.xml (deflated 40%) 2022-03-17T21:37:01.6557305Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212428.xml (deflated 40%) 2022-03-17T21:37:01.6557731Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212432.xml (deflated 40%) 2022-03-17T21:37:01.6558155Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212438.xml (deflated 41%) 2022-03-17T21:37:01.6558583Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212444.xml (deflated 41%) 2022-03-17T21:37:01.6559014Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212459.xml (deflated 41%) 2022-03-17T21:37:01.6559446Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212506.xml (deflated 41%) 2022-03-17T21:37:01.6559857Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212512.xml (deflated 41%) 2022-03-17T21:37:01.6560289Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212518.xml (deflated 41%) 2022-03-17T21:37:01.6560718Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212524.xml (deflated 40%) 2022-03-17T21:37:01.6561145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212530.xml (deflated 42%) 2022-03-17T21:37:01.6561574Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212537.xml (deflated 42%) 2022-03-17T21:37:01.6562001Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212543.xml (deflated 40%) 2022-03-17T21:37:01.6562413Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212549.xml (deflated 41%) 2022-03-17T21:37:01.6562902Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212555.xml (deflated 41%) 2022-03-17T21:37:01.6563329Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212601.xml (deflated 41%) 2022-03-17T21:37:01.6563758Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212607.xml (deflated 40%) 2022-03-17T21:37:01.6564190Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212613.xml (deflated 40%) 2022-03-17T21:37:01.6564620Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212619.xml (deflated 41%) 2022-03-17T21:37:01.6565047Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212625.xml (deflated 41%) 2022-03-17T21:37:01.6565466Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212631.xml (deflated 41%) 2022-03-17T21:37:01.6565894Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212836.xml (deflated 41%) 2022-03-17T21:37:01.6566324Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212842.xml (deflated 41%) 2022-03-17T21:37:01.6566795Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212848.xml (deflated 41%) 2022-03-17T21:37:01.6567232Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212852.xml (deflated 41%) 2022-03-17T21:37:01.6567660Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212856.xml (deflated 41%) 2022-03-17T21:37:01.6568091Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212902.xml (deflated 41%) 2022-03-17T21:37:01.6568504Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212908.xml (deflated 40%) 2022-03-17T21:37:01.6568925Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212912.xml (deflated 41%) 2022-03-17T21:37:01.6569357Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212917.xml (deflated 41%) 2022-03-17T21:37:01.6569786Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212921.xml (deflated 43%) 2022-03-17T21:37:01.6570214Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212922.xml (deflated 41%) 2022-03-17T21:37:01.6570646Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212927.xml (deflated 42%) 2022-03-17T21:37:01.6571057Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212931.xml (deflated 40%) 2022-03-17T21:37:01.6571486Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212935.xml (deflated 40%) 2022-03-17T21:37:01.6571917Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212939.xml (deflated 41%) 2022-03-17T21:37:01.6572345Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212944.xml (deflated 40%) 2022-03-17T21:37:01.6572772Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212948.xml (deflated 41%) 2022-03-17T21:37:01.6573260Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212952.xml (deflated 40%) 2022-03-17T21:37:01.6573690Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317212957.xml (deflated 41%) 2022-03-17T21:37:01.6574101Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213001.xml (deflated 41%) 2022-03-17T21:37:01.6574528Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213007.xml (deflated 41%) 2022-03-17T21:37:01.6574961Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213011.xml (deflated 41%) 2022-03-17T21:37:01.6575391Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213015.xml (deflated 41%) 2022-03-17T21:37:01.6575818Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213020.xml (deflated 41%) 2022-03-17T21:37:01.6576252Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213024.xml (deflated 41%) 2022-03-17T21:37:01.6576681Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213046.xml (deflated 41%) 2022-03-17T21:37:01.6577092Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213108.xml (deflated 43%) 2022-03-17T21:37:01.6577576Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213113.xml (deflated 41%) 2022-03-17T21:37:01.6578020Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213119.xml (deflated 40%) 2022-03-17T21:37:01.6578448Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213123.xml (deflated 41%) 2022-03-17T21:37:01.6578876Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213128.xml (deflated 43%) 2022-03-17T21:37:01.6579300Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213132.xml (deflated 42%) 2022-03-17T21:37:01.6579711Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213133.xml (deflated 42%) 2022-03-17T21:37:01.6580145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213134.xml (deflated 42%) 2022-03-17T21:37:01.6580879Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213136.xml (deflated 42%) 2022-03-17T21:37:01.6581311Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213137.xml (deflated 43%) 2022-03-17T21:37:01.6581747Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213138.xml (deflated 42%) 2022-03-17T21:37:01.6582175Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213139.xml (deflated 42%) 2022-03-17T21:37:01.6582605Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213140.xml (deflated 42%) 2022-03-17T21:37:01.6583023Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213141.xml (deflated 40%) 2022-03-17T21:37:01.6583453Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213146.xml (deflated 41%) 2022-03-17T21:37:01.6583880Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213150.xml (deflated 42%) 2022-03-17T21:37:01.6584401Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213151.xml (deflated 43%) 2022-03-17T21:37:01.6584832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213152.xml (deflated 41%) 2022-03-17T21:37:01.6585266Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213158.xml (deflated 41%) 2022-03-17T21:37:01.6585683Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213204.xml (deflated 40%) 2022-03-17T21:37:01.6586115Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213211.xml (deflated 40%) 2022-03-17T21:37:01.6586540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213217.xml (deflated 41%) 2022-03-17T21:37:01.6586974Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213221.xml (deflated 41%) 2022-03-17T21:37:01.6587405Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213225.xml (deflated 41%) 2022-03-17T21:37:01.6587834Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213229.xml (deflated 41%) 2022-03-17T21:37:01.6588321Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213233.xml (deflated 42%) 2022-03-17T21:37:01.6588748Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213238.xml (deflated 42%) 2022-03-17T21:37:01.6589178Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213242.xml (deflated 42%) 2022-03-17T21:37:01.6589612Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213246.xml (deflated 42%) 2022-03-17T21:37:01.6590039Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213250.xml (deflated 41%) 2022-03-17T21:37:01.6590466Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213254.xml (deflated 41%) 2022-03-17T21:37:01.6590893Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213259.xml (deflated 43%) 2022-03-17T21:37:01.6591324Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213300.xml (deflated 42%) 2022-03-17T21:37:01.6591737Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213304.xml (deflated 41%) 2022-03-17T21:37:01.6592165Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213308.xml (deflated 42%) 2022-03-17T21:37:01.6592596Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213309.xml (deflated 43%) 2022-03-17T21:37:01.6593024Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213311.xml (deflated 41%) 2022-03-17T21:37:01.6593453Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213315.xml (deflated 41%) 2022-03-17T21:37:01.6593883Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213319.xml (deflated 41%) 2022-03-17T21:37:01.6594295Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213323.xml (deflated 41%) 2022-03-17T21:37:01.6594723Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213327.xml (deflated 41%) 2022-03-17T21:37:01.6595213Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213329.xml (deflated 41%) 2022-03-17T21:37:01.6595644Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213330.xml (deflated 41%) 2022-03-17T21:37:01.6596071Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213334.xml (deflated 42%) 2022-03-17T21:37:01.6596502Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213338.xml (deflated 41%) 2022-03-17T21:37:01.6596930Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213342.xml (deflated 41%) 2022-03-17T21:37:01.6597340Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213347.xml (deflated 40%) 2022-03-17T21:37:01.6597771Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213351.xml (deflated 41%) 2022-03-17T21:37:01.6598201Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213355.xml (deflated 41%) 2022-03-17T21:37:01.6598629Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213359.xml (deflated 41%) 2022-03-17T21:37:01.6599099Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213404.xml (deflated 42%) 2022-03-17T21:37:01.6599536Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213405.xml (deflated 41%) 2022-03-17T21:37:01.6599960Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213406.xml (deflated 41%) 2022-03-17T21:37:01.6600375Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213407.xml (deflated 41%) 2022-03-17T21:37:01.6600800Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213411.xml (deflated 41%) 2022-03-17T21:37:01.6601221Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213416.xml (deflated 41%) 2022-03-17T21:37:01.6601647Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213420.xml (deflated 41%) 2022-03-17T21:37:01.6602076Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213424.xml (deflated 41%) 2022-03-17T21:37:01.6602505Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213428.xml (deflated 41%) 2022-03-17T21:37:01.6602920Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213434.xml (deflated 41%) 2022-03-17T21:37:01.6603348Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213440.xml (deflated 41%) 2022-03-17T21:37:01.6603776Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213444.xml (deflated 41%) 2022-03-17T21:37:01.6604206Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213450.xml (deflated 41%) 2022-03-17T21:37:01.6604636Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213456.xml (deflated 40%) 2022-03-17T21:37:01.6605067Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220317213506.xml (deflated 40%) 2022-03-17T21:37:01.6605533Z adding: test/test-reports/dist-gloo/distributed.test_launcher/TEST-TestDistributedLaunch-20220317213517.xml (deflated 46%) 2022-03-17T21:37:01.6605885Z adding: test/test-reports/dist-gloo/distributed.test_nccl/TEST-TestNCCLCUDA-20220317213523.xml (deflated 83%) 2022-03-17T21:37:01.6606251Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-FileStoreTest-20220317213527.xml (deflated 40%) 2022-03-17T21:37:01.6606621Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-FileStoreTest-20220317213529.xml (deflated 40%) 2022-03-17T21:37:01.6607021Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PrefixFileStoreTest-20220317213532.xml (deflated 40%) 2022-03-17T21:37:01.6607413Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PrefixFileStoreTest-20220317213534.xml (deflated 40%) 2022-03-17T21:37:01.6607810Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PrefixTCPStoreTest-20220317213537.xml (deflated 40%) 2022-03-17T21:37:01.6608192Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PrefixTCPStoreTest-20220317213539.xml (deflated 39%) 2022-03-17T21:37:01.6608571Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-PythonStoreTest-20220317213542.xml (deflated 40%) 2022-03-17T21:37:01.6608954Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-RendezvousEnvTest-20220317213544.xml (deflated 40%) 2022-03-17T21:37:01.6609339Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-RendezvousFileTest-20220317213547.xml (deflated 40%) 2022-03-17T21:37:01.6609771Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-RendezvousFileTest-20220317213549.xml (deflated 39%) 2022-03-17T21:37:01.6610154Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-RendezvousTest-20220317213552.xml (deflated 39%) 2022-03-17T21:37:01.6610503Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213555.xml (deflated 39%) 2022-03-17T21:37:01.6610874Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213557.xml (deflated 39%) 2022-03-17T21:37:01.6611231Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213600.xml (deflated 38%) 2022-03-17T21:37:01.6611592Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213602.xml (deflated 38%) 2022-03-17T21:37:01.6611949Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213605.xml (deflated 38%) 2022-03-17T21:37:01.6612309Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213607.xml (deflated 39%) 2022-03-17T21:37:01.6612653Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213610.xml (deflated 39%) 2022-03-17T21:37:01.6613022Z adding: test/test-reports/dist-gloo/distributed.test_store/TEST-TCPStoreTest-20220317213615.xml (deflated 39%) 2022-03-17T21:37:01.6613344Z adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 72%) 2022-03-17T21:37:01.6613664Z adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 71%) 2022-03-17T21:37:01.6613983Z adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 80%) 2022-03-17T21:37:01.6614331Z adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 80%) 2022-03-17T21:37:01.6614654Z adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 80%) 2022-03-17T21:37:01.6615028Z adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 68%) 2022-03-17T21:37:01.6615306Z adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%) 2022-03-17T21:37:01.6647641Z ##[group]Run seemethere/upload-artifact-s3@v3 2022-03-17T21:37:01.6647762Z with: 2022-03-17T21:37:01.6647889Z retention-days: 14 2022-03-17T21:37:01.6648016Z if-no-files-found: error 2022-03-17T21:37:01.6648225Z path: test-reports-*.zip 2022-03-17T21:37:01.6648328Z name: artifact 2022-03-17T21:37:01.6648447Z s3-bucket: gha-artifacts 2022-03-17T21:37:01.6648555Z region: us-east-1 2022-03-17T21:37:01.6648645Z env: 2022-03-17T21:37:01.6648811Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:01.6649111Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:01.6649287Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:01.6649502Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:01.6649619Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:01.6649696Z IN_CI: 1 2022-03-17T21:37:01.6649790Z IS_GHA: 1 2022-03-17T21:37:01.6649920Z IN_WHEEL_TEST: 1 2022-03-17T21:37:01.6650096Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:01.6650291Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:01.6650402Z PR_LABELS: [] 2022-03-17T21:37:01.6650669Z GITHUB_TOKEN: *** 2022-03-17T21:37:01.6650799Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:01.6650882Z PR_NUMBER: 2022-03-17T21:37:01.6651024Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:01.6651143Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:01.6651254Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:01.6651605Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:01.6651788Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:01.6651885Z TEST_CONFIG: distributed 2022-03-17T21:37:01.6651988Z SHARD_NUMBER: 1 2022-03-17T21:37:01.6652098Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:01.6652197Z PR_BODY: 2022-03-17T21:37:01.6652308Z GPU_FLAG: --gpus all 2022-03-17T21:37:01.6652388Z SHM_SIZE: 2g 2022-03-17T21:37:01.6652491Z ##[endgroup] 2022-03-17T21:37:02.0680741Z With the provided path, there will be 1 file uploaded 2022-03-17T21:37:02.0681311Z Uploading to s3 prefix: pytorch/pytorch/2000264336/artifact 2022-03-17T21:37:02.0727367Z Starting upload of test-reports-test_distributed_1_1-distributed-1-1-linux.8xlarge.nvidia.gpu.zip 2022-03-17T21:37:02.2841721Z Finished upload of test-reports-test_distributed_1_1-distributed-1-1-linux.8xlarge.nvidia.gpu.zip 2022-03-17T21:37:02.2996465Z ##[group]Run set -x 2022-03-17T21:37:02.2996854Z set -x 2022-03-17T21:37:02.2997205Z python3 -m pip install -r requirements.txt 2022-03-17T21:37:02.2997491Z python3 -m pip install boto3==1.19.12 2022-03-17T21:37:02.2997905Z GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}") 2022-03-17T21:37:02.2998277Z export GHA_WORKFLOW_JOB_ID 2022-03-17T21:37:02.2998662Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-03-17T21:37:02.3012040Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2022-03-17T21:37:02.3012341Z env: 2022-03-17T21:37:02.3012633Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:02.3013143Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:02.3013637Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:02.3014090Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:02.3014454Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:02.3014701Z IN_CI: 1 2022-03-17T21:37:02.3014901Z IS_GHA: 1 2022-03-17T21:37:02.3015129Z IN_WHEEL_TEST: 1 2022-03-17T21:37:02.3015413Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:02.3015801Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:02.3016123Z PR_LABELS: [] 2022-03-17T21:37:02.3016533Z GITHUB_TOKEN: *** 2022-03-17T21:37:02.3016802Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:02.3017195Z PR_NUMBER: 2022-03-17T21:37:02.3017470Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:02.3017758Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:02.3018013Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:02.3018523Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:02.3019059Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:02.3019381Z TEST_CONFIG: distributed 2022-03-17T21:37:02.3019631Z SHARD_NUMBER: 1 2022-03-17T21:37:02.3019852Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:02.3020086Z PR_BODY: 2022-03-17T21:37:02.3020649Z GPU_FLAG: --gpus all 2022-03-17T21:37:02.3020897Z SHM_SIZE: 2g 2022-03-17T21:37:02.3021114Z BRANCH: master 2022-03-17T21:37:02.3021340Z TAG: 2022-03-17T21:37:02.3021552Z WORKFLOW_ID: 2000264336 2022-03-17T21:37:02.3021793Z ##[endgroup] 2022-03-17T21:37:02.3050163Z + python3 -m pip install -r requirements.txt 2022-03-17T21:37:02.6006106Z Defaulting to user installation because normal site-packages is not writeable 2022-03-17T21:37:02.6303163Z Ignoring dataclasses: markers 'python_version < "3.7"' don't match your environment 2022-03-17T21:37:02.6870123Z Collecting astunparse 2022-03-17T21:37:02.7032166Z Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB) 2022-03-17T21:37:02.7401410Z Collecting expecttest 2022-03-17T21:37:02.7455271Z Downloading expecttest-0.1.3-py3-none-any.whl (6.5 kB) 2022-03-17T21:37:02.7544133Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2) 2022-03-17T21:37:03.3960437Z Collecting numpy 2022-03-17T21:37:03.4008685Z Downloading numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB) 2022-03-17T21:37:04.0113610Z Collecting psutil 2022-03-17T21:37:04.0176394Z Downloading psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB) 2022-03-17T21:37:04.1471371Z Collecting pyyaml 2022-03-17T21:37:04.1561824Z Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB) 2022-03-17T21:37:04.2818003Z Collecting requests 2022-03-17T21:37:04.2862617Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2022-03-17T21:37:04.3142759Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (49.1.3) 2022-03-17T21:37:04.3739409Z Collecting six 2022-03-17T21:37:04.3780596Z Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2022-03-17T21:37:04.4090181Z Collecting types-dataclasses 2022-03-17T21:37:04.4136216Z Downloading types_dataclasses-0.6.4-py3-none-any.whl (2.7 kB) 2022-03-17T21:37:04.4231349Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (3.10.0.2) 2022-03-17T21:37:04.5007106Z Collecting wheel<1.0,>=0.23.0 2022-03-17T21:37:04.5050499Z Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB) 2022-03-17T21:37:04.6026397Z Collecting urllib3<1.27,>=1.21.1 2022-03-17T21:37:04.6068198Z Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB) 2022-03-17T21:37:04.6712799Z Collecting idna<4,>=2.5; python_version >= "3" 2022-03-17T21:37:04.6753698Z Downloading idna-3.3-py3-none-any.whl (61 kB) 2022-03-17T21:37:04.7295428Z Collecting certifi>=2017.4.17 2022-03-17T21:37:04.7336984Z Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB) 2022-03-17T21:37:04.7919388Z Collecting charset-normalizer~=2.0.0; python_version >= "3" 2022-03-17T21:37:04.7959571Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2022-03-17T21:37:04.9112712Z Installing collected packages: six, wheel, astunparse, expecttest, numpy, psutil, pyyaml, urllib3, idna, certifi, charset-normalizer, requests, types-dataclasses 2022-03-17T21:37:04.9512060Z WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-03-17T21:37:04.9512792Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-03-17T21:37:06.9274633Z WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-03-17T21:37:06.9275354Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-03-17T21:37:07.3420602Z WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH. 2022-03-17T21:37:07.3421378Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2022-03-17T21:37:07.4040685Z Successfully installed astunparse-1.6.3 certifi-2021.10.8 charset-normalizer-2.0.12 expecttest-0.1.3 idna-3.3 numpy-1.21.5 psutil-5.9.0 pyyaml-6.0 requests-2.27.1 six-1.16.0 types-dataclasses-0.6.4 urllib3-1.26.9 wheel-0.37.1 2022-03-17T21:37:07.4616723Z + python3 -m pip install boto3==1.19.12 2022-03-17T21:37:07.7579469Z Defaulting to user installation because normal site-packages is not writeable 2022-03-17T21:37:08.5520719Z Collecting boto3==1.19.12 2022-03-17T21:37:08.5710687Z Downloading boto3-1.19.12-py3-none-any.whl (131 kB) 2022-03-17T21:37:08.6230720Z Collecting jmespath<1.0.0,>=0.7.1 2022-03-17T21:37:08.6285037Z Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2022-03-17T21:37:08.6758978Z Collecting s3transfer<0.6.0,>=0.5.0 2022-03-17T21:37:08.6800493Z Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2022-03-17T21:37:09.6851742Z Collecting botocore<1.23.0,>=1.22.12 2022-03-17T21:37:09.6907762Z Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB) 2022-03-17T21:37:09.9932155Z Collecting python-dateutil<3.0.0,>=2.1 2022-03-17T21:37:09.9978689Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2022-03-17T21:37:10.0177384Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.9) 2022-03-17T21:37:10.0391223Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0) 2022-03-17T21:37:10.1394986Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3 2022-03-17T21:37:11.0375892Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 2022-03-17T21:37:11.0902753Z ++ python3 .github/scripts/get_workflow_job_id.py 2000264336 i-038ed9ad29e72a29d 2022-03-17T21:37:11.4639881Z + GHA_WORKFLOW_JOB_ID=5590723920 2022-03-17T21:37:11.4640344Z + export GHA_WORKFLOW_JOB_ID 2022-03-17T21:37:11.4640986Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test 2022-03-17T21:37:16.2811940Z [scribe] Scribe access token not provided, sending report via boto3... 2022-03-17T21:37:16.2812533Z 2022-03-17T21:37:16.2812930Z ----- Historic stats comparison result ------ 2022-03-17T21:37:16.2813143Z 2022-03-17T21:37:16.2813393Z job: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:16.2813646Z commit: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:16.2813837Z 2022-03-17T21:37:16.2814048Z Commit graph (base is most recent master ancestor with at least one S3 report): 2022-03-17T21:37:16.2814291Z 2022-03-17T21:37:16.2814373Z : (master) 2022-03-17T21:37:16.2814588Z | 2022-03-17T21:37:16.2814846Z * 4b4f652f79 (HEAD) total time 4778.18s 2022-03-17T21:37:16.2815281Z * 96ed3320f8 (base) 3 reports, total time 4756.20s ± 1218.61s 2022-03-17T21:37:16.2815709Z * 119df90ead 3 reports, total time 4774.53s ± 1234.37s 2022-03-17T21:37:16.2816109Z * d8c0f2f35d 3 reports, total time 4760.99s ± 1153.41s 2022-03-17T21:37:16.2816530Z * 40a51edca7 3 reports, total time 4846.68s ± 1410.38s 2022-03-17T21:37:16.2817160Z * f7fe75c766 0 reports 2022-03-17T21:37:16.2817423Z * 79c3bbceb2 0 reports 2022-03-17T21:37:16.2817811Z * 14dcb5a1a0 3 reports, total time 4735.93s ± 1682.36s 2022-03-17T21:37:16.2818209Z * abc868dd64 3 reports, total time 4761.16s ± 1186.65s 2022-03-17T21:37:16.2818623Z * fafa54a940 3 reports, total time 4772.61s ± 1132.87s 2022-03-17T21:37:16.2819028Z * d67a265881 3 reports, total time 4769.61s ± 1158.61s 2022-03-17T21:37:16.2819292Z | 2022-03-17T21:37:16.2819497Z : 2022-03-17T21:37:16.2819614Z 2022-03-17T21:37:16.2819774Z Removed (across 567 suites) 0 tests, totaling 0.00s 2022-03-17T21:37:16.2820114Z Modified (across 0 suites) 0 tests, totaling 0.00s 2022-03-17T21:37:16.2820836Z Added (across 114 suites) 1530 tests, totaling +6620.27s 2022-03-17T21:37:16.3376948Z ##[group]Run .github/scripts/wait_for_ssh_to_drain.sh 2022-03-17T21:37:16.3377337Z .github/scripts/wait_for_ssh_to_drain.sh 2022-03-17T21:37:16.3391131Z shell: /usr/bin/bash -e {0} 2022-03-17T21:37:16.3391387Z env: 2022-03-17T21:37:16.3391681Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:16.3392187Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:16.3392684Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:16.3393087Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:16.3393449Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:16.3393675Z IN_CI: 1 2022-03-17T21:37:16.3393892Z IS_GHA: 1 2022-03-17T21:37:16.3394101Z IN_WHEEL_TEST: 1 2022-03-17T21:37:16.3394405Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:16.3394832Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:16.3395140Z PR_LABELS: [] 2022-03-17T21:37:16.3395566Z GITHUB_TOKEN: *** 2022-03-17T21:37:16.3395825Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:16.3396082Z PR_NUMBER: 2022-03-17T21:37:16.3396355Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:16.3396622Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:16.3396894Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:16.3397381Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:16.3397923Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:16.3398244Z TEST_CONFIG: distributed 2022-03-17T21:37:16.3398478Z SHARD_NUMBER: 1 2022-03-17T21:37:16.3398716Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:16.3398935Z PR_BODY: 2022-03-17T21:37:16.3399172Z GPU_FLAG: --gpus all 2022-03-17T21:37:16.3399396Z SHM_SIZE: 2g 2022-03-17T21:37:16.3399629Z ##[endgroup] 2022-03-17T21:37:16.3441351Z Holding runner for 2 hours until all ssh sessions have logged out 2022-03-17T21:37:16.3503275Z ##[group]Run # Ensure the working directory gets chowned back to the current user 2022-03-17T21:37:16.3503798Z # Ensure the working directory gets chowned back to the current user 2022-03-17T21:37:16.3504189Z docker run --rm -v "$(pwd)":/v -w /v "${ALPINE_IMAGE}" chown -R "$(id -u):$(id -g)" . 2022-03-17T21:37:16.3516257Z shell: /usr/bin/bash -e {0} 2022-03-17T21:37:16.3516510Z env: 2022-03-17T21:37:16.3516797Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:16.3517305Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:16.3517781Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:16.3518204Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:16.3518561Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:16.3518786Z IN_CI: 1 2022-03-17T21:37:16.3518999Z IS_GHA: 1 2022-03-17T21:37:16.3519251Z IN_WHEEL_TEST: 1 2022-03-17T21:37:16.3519543Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:16.3520071Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:16.3520396Z PR_LABELS: [] 2022-03-17T21:37:16.3520768Z GITHUB_TOKEN: *** 2022-03-17T21:37:16.3521037Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:16.3521297Z PR_NUMBER: 2022-03-17T21:37:16.3521554Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:16.3521847Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:16.3522104Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:16.3522610Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:16.3523221Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:16.3523529Z TEST_CONFIG: distributed 2022-03-17T21:37:16.3523784Z SHARD_NUMBER: 1 2022-03-17T21:37:16.3524006Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:16.3524255Z PR_BODY: 2022-03-17T21:37:16.3524484Z GPU_FLAG: --gpus all 2022-03-17T21:37:16.3524726Z SHM_SIZE: 2g 2022-03-17T21:37:16.3524943Z ##[endgroup] 2022-03-17T21:37:18.5187910Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2022-03-17T21:37:18.5188391Z # ignore expansion of "docker ps -q" since it could be empty 2022-03-17T21:37:18.5188744Z # shellcheck disable=SC2046 2022-03-17T21:37:18.5189035Z docker stop $(docker ps -q) || true 2022-03-17T21:37:18.5189350Z # Prune all of the docker images 2022-03-17T21:37:18.5189633Z docker system prune -af 2022-03-17T21:37:18.5202892Z shell: /usr/bin/bash -e {0} 2022-03-17T21:37:18.5203126Z env: 2022-03-17T21:37:18.5203441Z BUILD_ENVIRONMENT: linux-xenial-cuda11.3-py3.7-gcc7 2022-03-17T21:37:18.5203939Z DOCKER_IMAGE_BASE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7 2022-03-17T21:37:18.5204438Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2022-03-17T21:37:18.5204881Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2022-03-17T21:37:18.5205272Z TORCH_CUDA_ARCH_LIST: 5.2 2022-03-17T21:37:18.5205507Z IN_CI: 1 2022-03-17T21:37:18.5205731Z IS_GHA: 1 2022-03-17T21:37:18.5205965Z IN_WHEEL_TEST: 1 2022-03-17T21:37:18.5206252Z CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts 2022-03-17T21:37:18.5206645Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine 2022-03-17T21:37:18.5206955Z PR_LABELS: [] 2022-03-17T21:37:18.5207374Z GITHUB_TOKEN: *** 2022-03-17T21:37:18.5207666Z AWS_DEFAULT_REGION: us-east-1 2022-03-17T21:37:18.5207892Z PR_NUMBER: 2022-03-17T21:37:18.5208170Z SHA1: 4b4f652f799c732a7280052e44a803a4cf6084de 2022-03-17T21:37:18.5208442Z PYTORCH_RETRY_TEST_CASES: 1 2022-03-17T21:37:18.5208722Z GIT_DEFAULT_BRANCH: master 2022-03-17T21:37:18.5209229Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:18.5209772Z JOB_BASE_NAME: linux-xenial-cuda11.3-py3.7-gcc7-test 2022-03-17T21:37:18.5210099Z TEST_CONFIG: distributed 2022-03-17T21:37:18.5210332Z SHARD_NUMBER: 1 2022-03-17T21:37:18.5210575Z NUM_TEST_SHARDS: 1 2022-03-17T21:37:18.5210796Z PR_BODY: 2022-03-17T21:37:18.5211038Z GPU_FLAG: --gpus all 2022-03-17T21:37:18.5211281Z SHM_SIZE: 2g 2022-03-17T21:37:18.5211499Z ##[endgroup] 2022-03-17T21:37:18.8557141Z 68188c1d6584 2022-03-17T21:37:19.3776742Z Deleted Containers: 2022-03-17T21:37:19.3777111Z 68188c1d65840e77017513044fc8915cd31339d136d8cd4b0d622137ce388559 2022-03-17T21:37:19.3777328Z 2022-03-17T21:37:23.8165642Z Deleted Images: 2022-03-17T21:37:23.8166279Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine:latest 2022-03-17T21:37:23.8166992Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine@sha256:def822f9851ca422481ec6fee59a9966f12b351c62ccb9aca841526ffaa9f748 2022-03-17T21:37:23.8167570Z deleted: sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec 2022-03-17T21:37:23.8168245Z deleted: sha256:b2d5eeeaba3a22b9b8aa97261957974a6bd65274ebd43e1d81d0a7b8b752b116 2022-03-17T21:37:23.8168969Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7:c3998772049b8a8ba52cedfa525c9feda04cb2c7 2022-03-17T21:37:23.8169892Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-cuda11.3-cudnn8-py3-gcc7@sha256:e95af236903b074012c7b9a7d7543a1a50421cedce14158d8cbd1649b571f253 2022-03-17T21:37:23.8170487Z deleted: sha256:4b378382d0704769faa22f354eebfd5b68fe47737ebbfad4f93fdfd6cebe1a39 2022-03-17T21:37:23.8170923Z deleted: sha256:58748d8c926bb8e26b3a396d2d4b9a3175435998a2380ccfdd20af06b60250ca 2022-03-17T21:37:23.8171340Z deleted: sha256:1cef4108dbb2156aa940f3932d5e051ea5017bd6007885abd02d46fc66846bbc 2022-03-17T21:37:23.8171753Z deleted: sha256:5fce1be486ad298627d3e233e90ea338e0e640857eba4f114eafb15b33eac46d 2022-03-17T21:37:23.8172194Z deleted: sha256:07e2e2e3a2d1ab5d803a79df151e4fd5ceafbad4ae84a577c7cfd54eabf5a753 2022-03-17T21:37:23.8172802Z deleted: sha256:8c27bcc40fee68be2abe6615ac2b9a3ab3cbcb27bc1dba858d4cc7b1c5ebd408 2022-03-17T21:37:23.8173255Z deleted: sha256:daff149218fed7180fe3125311c62971ab27575a0456c38ac84bf278e00950bb 2022-03-17T21:37:23.8173681Z deleted: sha256:deb152643ed3597aca8ba4164674d6e4f9d9c68af27ca57f5e2f59a273beee61 2022-03-17T21:37:23.8174080Z deleted: sha256:72115cd67079969b9d277aa60e4544f3af09b0b89f1fc60a10b72fa2c9e7838b 2022-03-17T21:37:23.8174500Z deleted: sha256:11eebc15fa7fdc5d1464e095fda2506c24142e4698e754adfba74dc7a2951270 2022-03-17T21:37:23.8174920Z deleted: sha256:e255026c43a793980be2b079a835b0f043832ea41c850c3f5fff06b5778bc2f3 2022-03-17T21:37:23.8175329Z deleted: sha256:d70fadc265c7840d06fb5d6c24ce895cabfcc2abf3022cb90a0b0826f757a1c2 2022-03-17T21:37:23.8175755Z deleted: sha256:2842f8c0b60de5f4d44ac8787617708c86c29c00f0387d9e6e86a11dab1cdb69 2022-03-17T21:37:23.8176143Z deleted: sha256:5436b1e8b2f0f759ece4703a179c3e13485d779b9f5bb302e05ae07c05f59422 2022-03-17T21:37:23.8176554Z deleted: sha256:c1e24267a50c831a645f60c4730206eb424e193dab36f1c4a16bad32b3cff8b2 2022-03-17T21:37:23.8176968Z deleted: sha256:6554d81ab3a26d95773b5ba24eceee70cf4bd79463184e254a6ab3eedb983140 2022-03-17T21:37:23.8177379Z deleted: sha256:44add9a3c4d8c698b8359607d2c3a641948c3c56eb7ab6ae77e0d16f9cd901d9 2022-03-17T21:37:23.8177809Z deleted: sha256:125e36be48fdc93fcfb202eb6ab75dfa74e63417be3eaa88f01a1b455dcd06cf 2022-03-17T21:37:23.8178236Z deleted: sha256:b7aed65305973dab7bdca1e21f4d8c7b296c28d717a5c197e1a3fde786fcf7eb 2022-03-17T21:37:23.8178652Z deleted: sha256:9da1c0bba533653bf9930059c4e81053c80988367b052c7057256bb5bda00ec2 2022-03-17T21:37:23.8179062Z deleted: sha256:080d2ae90bd319703386d0e9bd41a3e9ca33a4db71bb0be9fa01fe9944e185f0 2022-03-17T21:37:23.8179465Z deleted: sha256:78f739db59986bb29cde26cd641d15070ebbb934d29168fa29f9bc034b4ce3ec 2022-03-17T21:37:23.8179868Z deleted: sha256:959667449e4cebf3d356a2f4f027437422f650ae1ad05790a520c9b10e2a3682 2022-03-17T21:37:23.8180562Z deleted: sha256:124589efa0f322dbf496ad9b60de18934894016a15d653cc37ea55c233153f1f 2022-03-17T21:37:23.8180994Z deleted: sha256:b02ef0d332840c6a45c17ad05d02e95091e8972ac624e1c67f91c475ab4ee59c 2022-03-17T21:37:23.8181420Z deleted: sha256:d51331dac47342d4c46de8df904dd0cac6df9e8b11f658d21dec69ff0d51fa67 2022-03-17T21:37:23.8181831Z deleted: sha256:d41830fd9c94ef302fa69b0f99adf9e658a7777ae7424e967718b0403feb066e 2022-03-17T21:37:23.8182257Z deleted: sha256:f265e1cb4ffd54c16b495c30df95b144dd1ad9804b79fab00b4e2f1b5a51cb6b 2022-03-17T21:37:23.8182663Z deleted: sha256:8cb1793c16fe3e2917e710c313e6c7f40356c57697afd1c869d1956af5529807 2022-03-17T21:37:23.8183076Z deleted: sha256:4b8971ad1bdb1784ce3d1632f1472ef82ae2c59985bdac7eae8b8ef35385bb42 2022-03-17T21:37:23.8183481Z deleted: sha256:861c18e75077780358417d2ebe6d82b38e9141f92b3be19866c3303d2872c3ee 2022-03-17T21:37:23.8183887Z deleted: sha256:ae944380e52611a5725d12094a03ead36f67fece91fa881f124b13f791decde4 2022-03-17T21:37:23.8184412Z deleted: sha256:d403a776bea8c034561014afb890a2e32a8fc7c2b056adbb4c4204114929dd23 2022-03-17T21:37:23.8184817Z deleted: sha256:f3e5730195690b47cda1dcd2b82b85e59f9d5128cd056369a59a7125d031d4b6 2022-03-17T21:37:23.8185232Z deleted: sha256:b2eca5673035ad2235f867bb08f248aad289c28b3e05c222c2bfa66065aff240 2022-03-17T21:37:23.8185653Z deleted: sha256:b8ea50d03fe6c1c0d49798d54fab146dc8599091d722fe89bc50917222a61cd9 2022-03-17T21:37:23.8186045Z deleted: sha256:2d97892293e049497ff07d8ac9f465b4ad820d6771d5026a3d2de8d0274c0173 2022-03-17T21:37:23.8186447Z deleted: sha256:cf4f95653407a97ab60b2e5637b6293ca842e1c79c20fadb152d8a03c3c52472 2022-03-17T21:37:23.8186862Z deleted: sha256:b1415f784c7c6824b4e63f7cdcda11fad9342077f9cd54215f186eb1a967c880 2022-03-17T21:37:23.8187254Z deleted: sha256:cb66a694646260150b66639650eccefd68bdb14b8574c6306c33f6062bb053e9 2022-03-17T21:37:23.8187642Z deleted: sha256:b1e53080c8b23a5d9619d24620053e8b57f2291344581b79a5a01f6c87d0a52e 2022-03-17T21:37:23.8188046Z deleted: sha256:d22d7bbe99aa69fc6acac585c47d74f3cb8a207161f0d75a42623e871c919494 2022-03-17T21:37:23.8188565Z deleted: sha256:7c9d162935b773e0eaee3543952a6cf7f736810ae252860c9a62af3721b3722f 2022-03-17T21:37:23.8188996Z deleted: sha256:d9e3660a0d54c230bb12c7644d658ddcc36a6c4c491357080bc8bc3b0ec9caca 2022-03-17T21:37:23.8189390Z deleted: sha256:78a354a2e014d645a01924f69bc2a6826330dff505cda42b453520c0077301a0 2022-03-17T21:37:23.8189785Z deleted: sha256:0214f4b057d78b44fd12702828152f67c0ce115f9346acc63acdf997cab7e7c8 2022-03-17T21:37:23.8190167Z deleted: sha256:1b9d0485372c5562fa614d5b35766f6c442539bcee9825a6e90d1158c3299a61 2022-03-17T21:37:23.8190572Z deleted: sha256:3c0f34be6eb98057c607b9080237cce0be0b86f52d51ba620dc018a3d421baea 2022-03-17T21:37:23.8190990Z deleted: sha256:be96a3f634de79f523f07c7e4e0216c28af45eb5776e7a6238a2392f71e01069 2022-03-17T21:37:23.8191214Z 2022-03-17T21:37:23.8304926Z Total reclaimed space: 14.23GB 2022-03-17T21:37:23.8371999Z Post job cleanup. 2022-03-17T21:37:23.9416893Z [command]/usr/bin/git version 2022-03-17T21:37:23.9466040Z git version 2.32.0 2022-03-17T21:37:23.9510017Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2022-03-17T21:37:23.9542684Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || : 2022-03-17T21:37:23.9866213Z Entering 'android/libs/fbjni' 2022-03-17T21:37:23.9905615Z Entering 'third_party/FP16' 2022-03-17T21:37:23.9945768Z Entering 'third_party/FXdiv' 2022-03-17T21:37:23.9985976Z Entering 'third_party/NNPACK' 2022-03-17T21:37:24.0026683Z Entering 'third_party/QNNPACK' 2022-03-17T21:37:24.0067113Z Entering 'third_party/XNNPACK' 2022-03-17T21:37:24.0118778Z Entering 'third_party/benchmark' 2022-03-17T21:37:24.0158026Z Entering 'third_party/breakpad' 2022-03-17T21:37:24.0199160Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T21:37:24.0241476Z Entering 'third_party/cpuinfo' 2022-03-17T21:37:24.0281932Z Entering 'third_party/cub' 2022-03-17T21:37:24.0322724Z Entering 'third_party/cudnn_frontend' 2022-03-17T21:37:24.0368155Z Entering 'third_party/eigen' 2022-03-17T21:37:24.0409329Z Entering 'third_party/fbgemm' 2022-03-17T21:37:24.0448261Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T21:37:24.0487148Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T21:37:24.0527328Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T21:37:24.0568454Z Entering 'third_party/flatbuffers' 2022-03-17T21:37:24.0610632Z Entering 'third_party/fmt' 2022-03-17T21:37:24.0650623Z Entering 'third_party/foxi' 2022-03-17T21:37:24.0689624Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T21:37:24.0729179Z Entering 'third_party/gloo' 2022-03-17T21:37:24.0768938Z Entering 'third_party/googletest' 2022-03-17T21:37:24.0808769Z Entering 'third_party/ideep' 2022-03-17T21:37:24.0847566Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T21:37:24.0889313Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T21:37:24.0936884Z Entering 'third_party/ios-cmake' 2022-03-17T21:37:24.0976177Z Entering 'third_party/kineto' 2022-03-17T21:37:24.1015053Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T21:37:24.1055226Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T21:37:24.1096294Z Entering 'third_party/nccl/nccl' 2022-03-17T21:37:24.1136353Z Entering 'third_party/neon2sse' 2022-03-17T21:37:24.1175373Z Entering 'third_party/onnx' 2022-03-17T21:37:24.1227671Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T21:37:24.1266841Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T21:37:24.1309825Z Entering 'third_party/onnx-tensorrt' 2022-03-17T21:37:24.1347914Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T21:37:24.1392858Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T21:37:24.1432374Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T21:37:24.1471479Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T21:37:24.1515959Z Entering 'third_party/pocketfft' 2022-03-17T21:37:24.1555060Z Entering 'third_party/protobuf' 2022-03-17T21:37:24.1597604Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T21:37:24.1636895Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T21:37:24.1679142Z Entering 'third_party/psimd' 2022-03-17T21:37:24.1718428Z Entering 'third_party/pthreadpool' 2022-03-17T21:37:24.1757483Z Entering 'third_party/pybind11' 2022-03-17T21:37:24.1797717Z Entering 'third_party/python-enum' 2022-03-17T21:37:24.1837377Z Entering 'third_party/python-peachpy' 2022-03-17T21:37:24.1878314Z Entering 'third_party/python-six' 2022-03-17T21:37:24.1917538Z Entering 'third_party/sleef' 2022-03-17T21:37:24.1957349Z Entering 'third_party/tbb' 2022-03-17T21:37:24.1999316Z Entering 'third_party/tensorpipe' 2022-03-17T21:37:24.2038942Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T21:37:24.2078463Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T21:37:24.2117390Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T21:37:24.2157936Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T21:37:24.2196810Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T21:37:24.2239390Z Entering 'third_party/zstd' 2022-03-17T21:37:24.2292859Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2022-03-17T21:37:24.2332312Z http.https://github.com/.extraheader 2022-03-17T21:37:24.2339628Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2022-03-17T21:37:24.2371573Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || : 2022-03-17T21:37:24.2687140Z Entering 'android/libs/fbjni' 2022-03-17T21:37:24.2710897Z http.https://github.com/.extraheader 2022-03-17T21:37:24.2742345Z Entering 'third_party/FP16' 2022-03-17T21:37:24.2765446Z http.https://github.com/.extraheader 2022-03-17T21:37:24.2797315Z Entering 'third_party/FXdiv' 2022-03-17T21:37:24.2820066Z http.https://github.com/.extraheader 2022-03-17T21:37:24.2851553Z Entering 'third_party/NNPACK' 2022-03-17T21:37:24.2875124Z http.https://github.com/.extraheader 2022-03-17T21:37:24.2907380Z Entering 'third_party/QNNPACK' 2022-03-17T21:37:24.2930411Z http.https://github.com/.extraheader 2022-03-17T21:37:24.2962259Z Entering 'third_party/XNNPACK' 2022-03-17T21:37:24.2986628Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3029851Z Entering 'third_party/benchmark' 2022-03-17T21:37:24.3052781Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3084457Z Entering 'third_party/breakpad' 2022-03-17T21:37:24.3107273Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3139299Z Entering 'third_party/breakpad/src/third_party/lss' 2022-03-17T21:37:24.3162250Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3195294Z Entering 'third_party/cpuinfo' 2022-03-17T21:37:24.3218070Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3249740Z Entering 'third_party/cub' 2022-03-17T21:37:24.3272556Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3303966Z Entering 'third_party/cudnn_frontend' 2022-03-17T21:37:24.3326880Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3364174Z Entering 'third_party/eigen' 2022-03-17T21:37:24.3388677Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3422455Z Entering 'third_party/fbgemm' 2022-03-17T21:37:24.3445858Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3477182Z Entering 'third_party/fbgemm/third_party/asmjit' 2022-03-17T21:37:24.3499863Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3531669Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2022-03-17T21:37:24.3554356Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3587040Z Entering 'third_party/fbgemm/third_party/googletest' 2022-03-17T21:37:24.3609692Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3642946Z Entering 'third_party/flatbuffers' 2022-03-17T21:37:24.3666072Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3699466Z Entering 'third_party/fmt' 2022-03-17T21:37:24.3723180Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3754871Z Entering 'third_party/foxi' 2022-03-17T21:37:24.3777513Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3808955Z Entering 'third_party/gemmlowp/gemmlowp' 2022-03-17T21:37:24.3831891Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3863572Z Entering 'third_party/gloo' 2022-03-17T21:37:24.3886365Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3918004Z Entering 'third_party/googletest' 2022-03-17T21:37:24.3940896Z http.https://github.com/.extraheader 2022-03-17T21:37:24.3972593Z Entering 'third_party/ideep' 2022-03-17T21:37:24.3996308Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4028057Z Entering 'third_party/ideep/mkl-dnn' 2022-03-17T21:37:24.4050313Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4082969Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN' 2022-03-17T21:37:24.4105691Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4143549Z Entering 'third_party/ios-cmake' 2022-03-17T21:37:24.4166376Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4198346Z Entering 'third_party/kineto' 2022-03-17T21:37:24.4221060Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4251724Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2022-03-17T21:37:24.4275391Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4306494Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2022-03-17T21:37:24.4328900Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4362334Z Entering 'third_party/nccl/nccl' 2022-03-17T21:37:24.4385977Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4417229Z Entering 'third_party/neon2sse' 2022-03-17T21:37:24.4440114Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4470989Z Entering 'third_party/onnx' 2022-03-17T21:37:24.4493599Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4537299Z Entering 'third_party/onnx/third_party/benchmark' 2022-03-17T21:37:24.4560906Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4592520Z Entering 'third_party/onnx/third_party/pybind11' 2022-03-17T21:37:24.4615208Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4649481Z Entering 'third_party/onnx-tensorrt' 2022-03-17T21:37:24.4672345Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4702777Z Entering 'third_party/onnx-tensorrt/third_party/onnx' 2022-03-17T21:37:24.4725202Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4761619Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark' 2022-03-17T21:37:24.4785294Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4816711Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11' 2022-03-17T21:37:24.4840375Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4871514Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang' 2022-03-17T21:37:24.4894809Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4931825Z Entering 'third_party/pocketfft' 2022-03-17T21:37:24.4955168Z http.https://github.com/.extraheader 2022-03-17T21:37:24.4986865Z Entering 'third_party/protobuf' 2022-03-17T21:37:24.5010732Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5045810Z Entering 'third_party/protobuf/third_party/benchmark' 2022-03-17T21:37:24.5069012Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5099961Z Entering 'third_party/protobuf/third_party/googletest' 2022-03-17T21:37:24.5123088Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5156190Z Entering 'third_party/psimd' 2022-03-17T21:37:24.5179747Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5210872Z Entering 'third_party/pthreadpool' 2022-03-17T21:37:24.5234345Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5265650Z Entering 'third_party/pybind11' 2022-03-17T21:37:24.5288332Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5319608Z Entering 'third_party/python-enum' 2022-03-17T21:37:24.5342606Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5373356Z Entering 'third_party/python-peachpy' 2022-03-17T21:37:24.5396853Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5428259Z Entering 'third_party/python-six' 2022-03-17T21:37:24.5451224Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5482622Z Entering 'third_party/sleef' 2022-03-17T21:37:24.5505939Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5537061Z Entering 'third_party/tbb' 2022-03-17T21:37:24.5559691Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5594099Z Entering 'third_party/tensorpipe' 2022-03-17T21:37:24.5616674Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5647817Z Entering 'third_party/tensorpipe/third_party/googletest' 2022-03-17T21:37:24.5670310Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5701202Z Entering 'third_party/tensorpipe/third_party/libnop' 2022-03-17T21:37:24.5724078Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5755623Z Entering 'third_party/tensorpipe/third_party/libuv' 2022-03-17T21:37:24.5777694Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5809109Z Entering 'third_party/tensorpipe/third_party/pybind11' 2022-03-17T21:37:24.5831695Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5861820Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2022-03-17T21:37:24.5884488Z http.https://github.com/.extraheader 2022-03-17T21:37:24.5918217Z Entering 'third_party/zstd' 2022-03-17T21:37:24.5941110Z http.https://github.com/.extraheader 2022-03-17T21:37:24.6185979Z Cleaning up orphan processes